Abstract
Orientation: Valid selection of master’s students is essential to the training and development of competent Industrial and Organisational Psychologists.
Research purpose: To validate a person-job fit data aggregation approach based on an integration of psychometric results in the selection of Industrial and Organisational Psychology master’s students.
Motivation for the study: The Employment Equity Act requires psychological assessment to be valid, reliable, culturally fair and based on the inherent requirements of the job. This mandates context-specific validation research, such as the current study.
Research approach/design and method: A non-experimental design was applied to secondary data (N = 133) derived from the Occupational Personality Questionnaire, Verify Ability Tests and academic success results of 5 student cohorts registered in the years 2012 to 2016. Data was analysed by means of descriptive and correlation statistics.
Main findings: The overall Person Job Match score did not show a statistically significant relationship with academic success. A significant relationship was observed between cognitive ability and academic success. Specific Person Job Match competencies presenting a significant relationship with academic success, contained scores from both personality and cognitive measures.
Practical implication: Ability tests should be weighted more strongly in selecting master’s students. A review of the master’s students’ competency profile may be needed to align with future world of work demands, and to improve its predictive role in academic success.
Contribution/value added: This study contributes to the predictive validity of the selection criteria for Industrial and Organisational Psychology master’s students.
Keywords: Academic success; VMG3; NMG3; OPQ32r; Competency-based assessment
Introduction
The future world of work poses several challenges to human resources and talent management, such as machine intelligence exceeding human capability, political uprising being driven online, virtual reality and artificial intelligence moving into the space of academia and research (Franklin, 2017; Spreitzer, Cameron, & Garrett, 2017; Xu, David, & Kim, 2018). It is predicted that over 26 billion devices will be connected through the internet of things (IoT) by the year 2020, replacing many current jobs and creating new ones (Morgan, 2014). The Fourth Industrial Revolution, or otherwise referred to as the digital revolution, is said to be the cause of this rapid change in the industry (Xu et al., 2018). Moreover, the uncertainty of the future world of work remains real (Omarjee, 2015) in the context of rising oil prices and the depletion of natural resources (Gratton, 2011; Maitland & Thomson, 2011).
In addition to global and technological changes impacting employees and the organisation, research scrutinising the changing face of the workforce points towards a slowly declining birth rate, an ageing workforce, a global lack of skilled employees and increased cultural diversity (Ganaie & Haque, 2017). As a result of the lack of highly skilled employees to face this technologically driven future world of work, organisations are at war for talent to remain competitive and relevant (Ganaie & Haque, 2017; Wilcox, 2016). Branson (2014) wrote that an organisation is only as strong as its people and that selecting talent and putting employees at the centre of the business are central to remaining viable and successful.
One discipline dedicated to providing evidence-based data to improve organisational effectiveness is the field of industrial and organisational (IO) psychology (Salas, Kozlowski, & Chen, 2017). The IO psychologist plays a vital role in helping organisations to understand the impact of the digital revolution that is currently disrupting every industry, in every country, along with the entire system of management, governance and production (Xu et al., 2018). The Society of Industrial and Organizational Psychology (SIOP), in the United States of America, defines the role of the IO psychologist as someone who works with individuals, organisations and society as a whole, understanding systemic interdependence and recognising the effects that political influences, consumers and skill shortages have on the future world of work (Landy & Conte, 2016). Industrial and organisational psychology is the scientific study of individual, group and organisational behaviour in the workplace and is aimed at optimising performance throughout (Cilliers & Flotman, 2016). The IO psychologist is additionally tasked to help his or her clients manage talent resources (Ganaie & Haque, 2017; Theron, 2009; Wilcox, 2016), ensuring a work environment that is focused on the well-being and individuality of the employee (Lawler, 2011; Van Vuuren, 2010) and on challenges that the employee faces outside of the work environment (Barkhuizen, Jorgensen, & Brink, 2014; Landy & Conte, 2016).
To register as an IO psychologist in South Africa, certain educational and professional registration requirements must be met (Health Professions Act, No. 56 of 1974). These include completion of relevant bachelors and honours degrees, a coursework master’s degree, a 12-month supervised internship and passing the profession’s Board of Psychology examination (Cilliers & Flotman, 2016; HPCSA, 2017). With relevance to this study, the coursework master’s degree is structured into two parts: one being the coursework year (referred to as the M1 year) and the second part is the M2 year that encompasses the dissertation section, spanning 1–3 years (Cilliers & Flotman, 2016; Viviers & Van Niekerk, 2012). The M1 year entails coursework covering career, personnel and organisational psychology, as well as psychometrics, psychological research and personal growth (Cilliers & Flotman, 2016). The M1 and M2 parts each contribute 50% towards the final mark for the degree (Cilliers & Flotman, 2016; Viviers & Van Niekerk, 2012). In their 2016 study, Cilliers and Flotman raised concerns that, even though most of the selected students in the IO psychology coursework master’s degree seemed to have passed the M1 part of the degree, only 60% of these students completed the M2 part of the programme, resulting in an overall lower throughput rate (cf. Cilliers & Harry, 2012).
To build capacity of qualified IO psychologists in the world of work, educational institutions need to select the best candidates to complete the qualification successfully, in a reasonable time frame at the standard of the institution (Taylor, McManus, & Davison, 2018). Having a valid selection procedure to select students who meet the requirements of the M1 and the M2 years successfully is, therefore, imperative. Using valid selection measures in South Africa is also a legal requirement constituted in the Employment Equity Act (EEA), No. 55 (1998).
Research purpose and objectives
The purpose of this study was to validate a person-job fit data aggregation approach applied in the selection of IO psychology master’s students. Such an approach is based on deriving a Person Job Match (PJM) score from an integration of results from cognitive and personality measures applied in the selection battery. The research objectives therefore included an investigation of the predictive validity of the PJM score that makes up one part of the final result of the competency-based assessment battery applied to select IO psychology master’s students, regarding their academic success.
The following research hypotheses were formulated to guide the study:
H0: The PJM score as a result of the IO psychology master’s selection assessment battery is not a valid predictor of academic success.
H1: The IO psychology master’s PJM score is a valid predictor of academic success.
H2: The Graduate Verify Verbal Ability Test (VMG3) and Graduate Verify Numerical Ability Test (NMG3) display a significant relationship with academic success.
H3: The Occupational Personality Questionnaire (OPQ32r) displays a significant relationship with academic success.
Literature review
Predictors of academic success
Common factors that are traditionally related to predicting academic success include the students’ socio-economic status and biographical details (Tinto, 1993), as well as the academic marks that they achieved during their schooling (Jeynes, 2016). Although these factors remain regarded as valid predictors, recent research points to a variety of factors relating to the prediction of academic success. These include the students’ interests and motivation, self-confidence, emotional intelligence, challenging themselves cognitively, conscientiousness, ethnicity, gender, reflective writing, stable role-models, parental support, communication skills, technological competence, support received and the general communication skills of the students and teachers (Adamiak & Sauls, 2017; Akgündüz & Akınoğlu, 2017; Bartram, 2005; Biçer, 2017; Grass, Strobel, & Strobel, 2017; Kale & Etyemez, 2017; Kappe & Van der Flier, 2011; Khan, Khan, Zia-Ul-Islam, & Khan, 2017; Killen, 1994, Kotzé & Griessel, 2008; Machika & Johnson, 2015; Önder & Şeyma, 2017; Parker, Saklofske, & Keefer, 2017; Roksa & Whitley, 2017; Salend & Whittaker, 2017; Schreiber & Yu, 2016; Schmidt & Hunter, 1998; Tsingos-Lucas, Bosnic-Anticevich, Schneider, & Smith, 2017).
Of importance to this study, in a meta-analysis of 19 selection methods taken over 85 years with the criteria being job and training performance, Schmidt and Hunter (1998) found job experience as the lowest predictor and biographical data, conscientiousness, integrity and cognitive ability amongst the most reliable predictors for academic success. Cognition far outranked the rest of the predictors in the meta-study (Bartram, 2005; Kotzé & Griessel, 2008). The general mental ability, or what is commonly referred to as a person’s intelligence, is the ‘most validated individual differentiating construct in psychology’ (Puchert, Dodd, & Viljoen, 2017, p. 1). Mathematics as a problem-solving ability has further been found to be a strong predictor for academic success and is motivated by Puchert et al. (2017) to be a critical part of the selection criteria for post-school training programmes. The study, additionally, adds that secondary training can be used as a useful pre-screening tool to admit candidates into higher level training programmes. In general, the predictive validity of personality measures has been widely documented (Dilchert, Ones, & Krueger, 2019; Van Aarde, Meiring, & Wiernik, 2017).
The inclusion of personality measures combined with measures of cognition is commonly used in selection for access to training programmes (Kappe & Van der Flier, 2011). In this regard, and in the context of the EEA (1998), a competency-based approach is imperative (Potgieter & Van der Merwe, 2002).
When cognitive and personality assessments are used to inform selection, which is the case in this study, the validity of the specific measurement outcomes, such as the PJM score, remains essential.
Validity
Psychometric measures form part of the competency-based assessment battery that is currently applied in selecting IO psychology master’s students at the research institution. Specifically, this includes a personality measure, the OPQ32r, and a cognitive measure, the Verify Ability Tests (VMG3 & NMG3), from which the PJM score is derived. As these psychological assessments are used for selection, specific legislative guidelines need to be adhered to according to the EEA, No. 55 (1998). The EEA (1998), in this case, has the dual objective of ensuring that only valid and reliable assessments are used, and that assessments are used in a fair manner, free from bias and unfair discrimination. In this legal context, it is critical that the PJM score that is derived from the psychometric measures used to select IO psychology master’s students’ be evaluated for its predictive validity in terms of academic success.
Psychometric measures are grounded in the validity of the measure used (Schmidt, 2006). The validity of a measure is concerned with how well the assessment measures what it is supposed to be measuring. There are three types of validities, namely, content, construct and criterion validities (Foxcroft & Roodt, 2009; Salkind, 2016). Content validity is a non-statistical method that concerns itself with whether the measure actually measures what it is designed to measure (Foxcroft & Roodt, 2009; Salkind, 2016). Construct validity refers to the theoretical construct of the measure and its validity (Roodt, 2009; Salkind, 2016). Lastly, criterion validity concerns itself with the relationship between the predictor(s) (in this study, the results of the psychometric measures) and the criterion (in this study, academic success) (Salkind, 2016). Salkind (2016) pointed towards two essential aspects to consider when conceptualising validity. One, validity refers to the results of the measure or test and not the measure or test itself. Two, the results show a degree of validity (not absolute validity) referring to the degree of relationship between the covariates and the criterion (Potter, 2006; Salkind, 2016). In this study, the validity degree of how well the PJM score predict academic success was investigated and significant relationships between the individual psychometric measures and academic success was explored (Foxcroft & Roodt, 2009; Kriek, 2009; Roodt, 2009; Salkind, 2016).
Fairness in selection: Competencies reflecting the inherent requirements of the job
Regarding fairness, the EEA (1998) stated that discrimination (relevant to selection) may be regarded as fair when it is aligned with the inherent requirements of the job. Therefore, when establishing an assessment battery, following a competency-based assessment framework is essential to ensure compliance with fair discrimination in the EEA (1998) (Potgieter & Van der Merwe, 2002).
The competency profile of an ideal IO psychology master’s student was created by conducting a job analysis of the IO psychologist’s role. This competency profile was designed by subject matter experts (SMEs), which consisted of practising IO psychologists and IO psychology master’s academic staff members (Viviers & Van Niekerk, 2012). SHL’s 20 generic competencies (SHL, 2009) were used as a point of departure, as it includes competencies that have been found to contribute to superior performance in different roles and positions in the workplace (SHL, 2011). The competencies were chosen according to their relevance with the competencies described by the SMEs, which, in turn, was guided by success criteria in the IO psychologist’s role identified by the Texas IO psychologists (TIOPs) (Blakeny et al., 2002) and the HPCSA’s professional training and registration requirements (Viviers & Van Niekerk, 2012).
The competency profile was organised along essential, desirable, less relevant and not relevant competencies. Essential competencies indicate those competencies identified by the SME panel to be required for nearly all the role objectives of an IO psychologist. Desirable competencies are those competencies identified to be crucial to most job requirements, whereas relevant competencies were those required in meeting at least some objectives. Less relevant and not relevant competencies were indicated by the SME panel to have very little relevance to the competency profile of the IO psychologist and were, therefore, excluded from the final competency profile (Viviers & Van Niekerk, 2012). Table 1 outlines the competencies in the competency profile of an IO psychology master’s student. The essential and desirable competencies consequently constituted the selection criteria against which applicants were assessed for entry into the IO psychology master’s programme at the research institution from the period 2012 to 2016.
| TABLE 1: SHL’s 20 generic competencies† included in the industrial and organisational psychology master’s competency profile. |
Research design
Research approach
In this non-experimental, positivist study, a descriptive quantitative approach was followed. Multivariate statistical techniques were used to analyse secondary data that were collected over a period of 5 years, to explain the relationships of the constructs and variables (Babbie, 2005). Secondary data were derived from the period 2012 to 2016 including the psychometric and academic results for IO psychology master’s students. The psychometric results included students’ Verify Graduate Ability Test scores, OPQ32r sten scores and the overall PJM competency profile scores. These scores were covariates assumed to be affecting the academic success of the IO psychology master’s student (cf. Potter, 2006).
The Verify Graduate Ability Test is a cognitive measure and the OPQ32r, a personality measure. The PJM competency profile is constructed on the essential and desirable competencies of an IO psychologist. Scores are calculated for each of these PJM competencies from the results of the Verify Graduate Ability Tests and the OPQ32r. The Verify Graduate Ability Tests’ sten scores, the OPQ32r sten scores, the PJM competency profile sten scores and the PJM overall scores were sourced from the responsible, independent test publisher. The students’ academic success scores were accessed from the student system of the research institution from the period 2012 to 2016. The academic results included students’ final results of the M1 year, the final results of the M2 year and the overall results of the completed degree.
Research method
Research participants
The sample (n = 133) consisted of the IO psychology master’s students who have been successfully selected into the IO psychology master’s programme from the years 2012 to 2016. The sample consisted of female students (72.2%) and males (27.8%). The white student cohort made up the largest number of students (40.6%) with the remainder being African (34.6%), Indian (15%) and mixed race (9.8%). The minimum requirement to apply for the IO psychology master’s degree is a related honours degree. Only 1.5% of the students had a degree or 3-year diploma before entering the IO psychology master’s programme, where the rest had honours degrees (98.5%). The few diploma and/or undergraduate students (1.5%) were because of them having been busy completing their honours degrees during the time of the data collection. The sample showed the following along the native language of the students: English (39.8%), Afrikaans (24.1%), Zulu (7.5%), Tswana (6%), Tsonga (5.3%), Unspecified (4.6%), Southern Sotho (2.3%), Xhosa (2.3%) and Venda and Ndebele at both (1.5%) each.
Measuring instruments
The data were collected on the IO psychology master’s students’ results on the Verify Graduate Ability Tests, the OPQ32r, the PJM competency profile scores, the PJM overall score and academic results from their M1 and M2 years and final overall master’s, in a 5-year period spanning from 2012 to 2016.
Verify graduate ability tests: SHL’s Verify Graduate Ability Tests were used to measure verbal (VMG3) and numerical (NMG3) reasoning. Both the numerical and verbal tests are deductive reasoning tests that work on the assumption that problems can be solved by applying previously established rules (Burke, Van Someren, Tatham, & Downey, 2013). The results of the Verify Graduate Ability Tests are presented in two transformed standard score scales, namely, a T-score and a sten score along which interpretations can be made (SHL, 2007). Studies based on both ability tests have proven to be predictable, significant and substantial in their correlations with academic success, with estimated validities ranging from 0.4 to 0.6 (Kotzé & Griessel, 2008; SHL, 2007). In terms of the reliability of the ability tests, Kotzé and Griessel (2008) reported high alpha coefficients (between 0.82 and 0.91). The norm group used for the Verify Graduate Ability Tests was the general UK population who had been normed for South African purposes (SHL, 2007).
Occupational personality questionnaire: The OPQ32r was developed based on the trait theory of personality and applies a forced-choice method that has normative properties (CEB, 2010).A benefit of the OPQ32r is that it is highly reliable (SHL, 2009) with the mean of the Item Response Theory (IRT) composite reliability being 0.84 (SHL, 2009). This personality measure provides information on the individual’s preferred style of behaviour at work and is commonly used for managerial and professional applications (SHL, 2009).
Thirty-two dimensions of an individual’s preferences about behaviour at work make up this occupational model of personality (Burke et al., 2013; CEB, 2010; Joubert, Ilke, Bartram, Dowdeswell, & Lin, 2015). The 32 dimensions are clustered in eight broad sub-domain scales and are also referred to as the ‘Great Eight’ competency factors that have been developed through self-rating and manager ratings of work performance. They are further categorised under three main categories, namely, relationships with people, thinking styles and feelings and emotions (Bartram, 2005, p. 1185).
The norm group applied in interpreting applicants’ OPQ32r results consisted of a large UK population who had been normed for South African purposes (SHL, 2009). According to the HPCSA’s list of classified and certified psychological assessments, the OPQ32r is registered as a psychological assessment (HPCSA, 2017).
The Person Job Match profile: The scores from two of the psychometric measures applied in the selection assessment were used to create a candidate’s PJM profile, weighted in accordance with the competency profile of an identified role (SHL, 2018).
The Verify Graduate Ability Tests (VMG3, NMG3) and the OPQ32r were used to calculate a competency potential score considering each of the desirable and essential competencies in which the IO psychology master’s student needs to prove competence for selection purposes. These scores constitute the candidate’s PJM profile and indicate the extent of fit between the applicant’s competencies and the required role competencies. An overall match score is also calculated, indicating an overall PJM score, as well as the bands of an extremely weak, weak, moderate, strong or extremely strong match (SHL, 2018). The PJM profile, thus, provides an overall score and a competency potential score for a candidate for each of the competencies in the profile (i.e. the essential and desirable competencies indicating the student’s level of suitability for the role of an IO psychology master’s student) (SHL, 2011).
All the essential and desirable competencies derive their PJM competency scores from the OPQ32r. Competencies in the PJM profile that specifically include results from the Verify Ability Tests are Presenting and Communicating, Writing and Reporting, Analysing and Learning and Researching (SHL, 2007).
Additional competencies that are not part of the IO psychology master’s students’ PJM profiles that include scores from the Verify Ability Tests are Applying Expertise and Technology, Creating and Innovative and Formulating Strategies and Concepts (SHL, 2007).
Statistical analyses
Descriptive statistics (means and standard deviations) were used to determine and describe the degree to which the variables exist in the sample. A Pearson’s two-tailed correlation was used to measure the level of strength between two variables (Lachenicht, 2002), namely, the students’ academic achievement scores (their M1, M2 and final percentage scores) and the sten scores of the Verify Graduate Ability Tests, the OPQ32r and the 12 competency scores of the PJM profile. Additional PJM competencies that are not part of the IO psychology master’s student’s competency profile were investigated with regard to possible significance to academic success. The statistical significance was set at 0.01 and 0.05. In terms of interpreting the size of the relationship, Cohen’s (1988) guidelines for practical significance were adopted, where a magnitude of r ≥ 0.100 is considered small, ≥ 0.300 is considered moderate and ≥ 0.500 is considered large. For the purpose of this study, the small, substantial, strong and dependable relationships are reported.
Ethical considerations
The initial instructions for the online assessments were sent to applicants via e-mail. The e-mail provided a link to the online assessment portal. Informed consent was obtained from each potential student, on the online assessment portal, before the applicant was able to proceed with the assessments. Applicants were informed of the purpose and nature of the assessment and that their results may be used for research purposes. Further ethical clearance was sought from the research institution to access the nonidentifiable data from the student management system. Ethical clearance to use the secondary data was obtained from the Research Ethics Review Committee of the relevant College at the research institution (ERC reference number: 2017_CEMS/IOP_010).
Results
Of the total sample (n = 133), a small number of IO psychology master’s students failed their M1 year (6.8%). As the M2 year can be completed on a full- or part-time basis (Viviers & Van Niekerk, 2012), a large number of student data were outstanding (n = 78). The outstanding data result from the number of IO psychology master’s students still in the process of completing their M2 year (n = 63) plus those who have dropped out (n = 15). Furthermore, no-fail data were available for the M2 year because the thesis that forms part of the M2 year only gets submitted once it has been thoroughly checked by the attending supervisors, at the research institution, before the IO psychology master’s student can submit it to be examined (Viviers & Van Niekerk, 2012). This means that up until 2016, only 41.4% (n = 55) of students registered from 2012 to 2016 had completed both the M1 and M2 years successfully.
Correlation statistics
Pearson’s two-tailed correlation is used to measure the level of strength between variables (Lachenicht, 2002).
The variables consisted of the Verify Graduate Ability Test sten scores, the OPQ32r sten scores, the PJM overall score and the PJM competency scores that were used as covariates affecting the dependent variable, namely, the academic success rates of the IO psychology master’s students (Potter, 2006). Academic success is seen as the percentage scores obtained in the M1 year, the M2 year and the final percentage score on completion of the degree. The M1 year consists of five compulsory course work modules, each for which a final mark is derived from the students’ formative and summative assessments for the module. The M1 percentage score is the average of the five course work modules’ final marks. The M2 percentage score is the average of the marks allocated to the student’s dissertation of limited scope by two examiners. The final overall percentage score is the average of the student’s M1 and M2 percentage scores. The results of the correlation analysis pertain to the set hypotheses and are discussed here in terms of whether they confirm the research hypotheses.
Hypothesis 1: The industrial and organisational psychology master’s person job match score is a valid predictor of academic success
Table 2 reports on the correlation coefficients of the overall PJM and PJM band scores with academic success. No statistically significant relationship was found between either the overall PJM scores or the PJM band scores.
| TABLE 2: Pearson’s correlation between the person job match overall score and band score and academic success. |
Table 3 reports the correlation coefficients of the IO psychology master’s student’s PJM competency profile with academic success. Individual competencies that have proven to display a statistically significant relationship of small effect size (Cohen, 1988) with academic success (p ≤ 0.05) include writing and reporting (r = 0.209) and coping with pressure and setbacks (r = -0.183) for the M1 year. A statistically significant relationship with academic success (p ≤ 0.01) was also found for competencies that include learning and researching (r = 0.238) and analysing (r = 0.232) for the M1 year. Individual competencies that have proven to display a statistically significant relationship with academic success in terms of the final result (p ≤ 0.05) include learning and researching (r = 0.288). Hypothesis 1 is not accepted.
| TABLE 3: Pearson correlation statistics: Essential and desirable person job match competency scores with academic success. |
For the sake of interest, Table 4 reports the correlation coefficients of the PJM competencies that are not included in the IO psychology master’s competency profile with academic success. Two of these competencies display a statistically significant relationship with academic success, namely, creating and innovating with the M1 year (r = 0.271) (p ≤ 0.01) and the final result (r = 0.324) (p ≤ 0.05) and formulating strategies and concepts with the M1 year (r = 0.173) (p ≤ 0.05).
| TABLE 4: Pearson’s correlation statistics: Less relevant and not relevant person job match competency scores with academic success. |
Hypothesis 2: The verify graduate ability test display a significant relationship with academic success
Table 5 shows the correlation coefficients of the Verify Graduate Ability Test with academic success. The correlation between the Verify Graduate Numerical Ability Test and academic success displays a correlation coefficient of 0.199 (p ≤ 0.05) for the M1 year, 0.323 (p ≤ 0.05) for the M2 year and 0.305 (p ≤ 0.05) for the overall result of the programme. In terms of Cohen’s (1988) guide to interpreting effect sizes, a definite, small to moderate relationship is evident, demonstrating the Verify Graduate Numerical Ability Test’s ability to predict academic success as seen in the M1 year, the M2 year and the overall programme results. The correlation between the Verify Graduate Verbal Ability Test and academic success displays a correlation coefficient of 0.218 (p ≤ 0.05) for the M1 year, -0.004 for the M2 year and 0.193 for the overall result of the programme. The magnitude of r is between 0.201 and 0.400 of the M1 year only. Therefore, a significant relationship of small effect size is evident, demonstrating the Verify Graduate Verbal Ability Test’s ability to predict academic success as seen in the M1 year only. Hypothesis 2 is accepted.
| TABLE 5: Pearson’s correlation between academic success and to verify graduate ability tests (verbal and numerical). |
Hypothesis 3: The occupational personality questionnaire displays a significant relationship with academic success
Table 6 reports on the correlation coefficients between the OPQ32r sten scores and academic success. Only a few individual competencies were found to display a statistically significant relationship with academic success on the lower score for the M1 year that includes outspoken (r = -0.198), relaxed (r = -0.217), optimistic (r = -0.175) and trusting (r = -0.192) (p ≤ 0.05) and on the higher score for emotionally controlled (r = 0.224) (p ≤ 0.01). The competency, innovative, saw a significant relationship on the lower score (r = -0.278) (p ≤ 0.05) with the M2 year results and worrying on the higher score (r = 0.298) (p ≤ 0.05) with the M2 year results of the IO psychology master’s students. Hypothesis 3 is not accepted because of the limited significant relationships of small effect size observed.
| TABLE 6: Pearson’s correlation between academic success and the occupational personality questionnaire sten scores. |
The null hypothesis which states that the PJM score of the IO psychology master’s competency profile is not a valid predictor of academic success is, therefore, not rejected. Only the Verify Graduate Ability Tests displayed significant relationships for the M1, M2 and final academic results of the IO psychology master’s student. The OPQ32r sten scores only displayed a select few competencies that display a small, but definite relationship with academic success. The overall PJM score and PJM band score did not display any relationship with predicting academic success, and of the PJM competencies used for the selection of the IO psychology master’s students, only a few displayed a significant relationship, which was mostly because of the fact that the Verify Graduate Ability Test has a weighting in the competency scores that displayed predictive values.
Discussion
The aim of this study was to investigate whether cognitive and personality-based psychometric results, aggregated into a person-job fit PJM score, predict academic success for a sample (n = 133) IO psychology master’s students registered between 2012 and 2016. An average number of 27 students made up the student cohort in each year. More than two-thirds of the sample was made up of female students and close to half of the sample consisted of white students. Not all students who had been selected displayed a strong to extremely strong match with regard to their PJM student profile results or band category. This could be because of the fact that the PJM band match only weighs 25% of the final recommendation for access to the programme and other selection criteria are considered such as, academic performance in the honours degree, an interview and a simulation exercise.
The overall academic success results of the IO psychology master’s students with regard to pass and fail percentage was only available for the M1 year, which displays a pass rate of 93.4% for the total sample (n = 133). This means that most students who were selected into the M1 year completed the academic requirements of the M1 year, whilst the academic requirements of the M2 year are incomplete for a large proportion of the sample. Of the students in the sample who started with the master’s programme, 15 students dropped out, 55 students completed their IO psychology master’s degree and 78 students were still in the process of completing their IO psychology master’s degree. The M2 year consists of a research component, where the student is tasked to complete a research dissertation of limited scope that can be completed over 2–4 years (Viviers & Van Niekerk, 2012). It is, therefore, acceptable that most of the students who completed their master’s degree were registered in the 2012–2014 period. Only a few students first registered in 2015 and 2016 completed their degrees at the time of the study. Yet, many students who had been registered for 4 or more years have not completed their degrees and mostly because the M2 year was not finalised. A study that focused on postgraduate throughput rates found that students who are busy with the research component of a master’s or doctoral degree are said to struggle with multiple roles, feelings of isolation and rigid programme structures (Abiddin, 2011). Suggestions along supporting students during their research section of the postgraduate degree include establishing regular meetings between supervisor and student and requesting regular progress reports from the students (Abiddin, 2011).
The Verify Graduate Ability Tests displayed a definite, but small relationship with academic success of the M1 year results. More specifically, the Verify Graduate Numerical Reasoning test specifically showed a definite, but small relationship with the M1, M2 and final results of the IO psychology master’s students. The Verify Graduate Verbal Reasoning test showed a definite but small relationship with the M1 marks only. The validity coefficients as found in this study (0.199–0.323) are not in line with Kotzé and Griessel’s (2008) study that found correlation coefficients from 0.400 to 0.600. This discrepancy may be influenced by the restriction of range on the ability tests, in that results were only available for those students selected into the program. Regardless, one could postulate that the Verify Graduate Ability Test displays the strongest relationship in this study as mirrored in various studies which suggest that cognition may be the best predictor for academic success (Bartram, 2005; Kappe & Van der Flier, 2011; Kotzé & Griessel, 2008; Puchert et al., 2017). Numerical reasoning is further motivated to be included as selection criteria into higher training because of its predictive values for academic success (Puchert et al., 2017).
The OPQ32r sten scores presented with minor statistical relevance to the criterion. Statistically significant relationships between select competencies and the M1 and M2 academic results can be highlighted. With significance levels with academic achievement, respectively (p ≥ 0.0005 and p ≥ 0.0001), the IO psychology master’s student who achieved academically, appears most likely to be reserved by holding back from criticising others and may not express his or her own views or own opinions (outspoken), may tend to be tenser (relaxed), nervous before significant events (worrying), concerned about the future (optimistic) and wary of others’ intentions (trusting) whilst displaying emotions clearly (emotionally controlled) (SHL, 1999). The limited relationship of the OPQ32r and academic success could be attributed to possible range restriction as the sample constituted only successfully selected students.
The correlational analysis found no relationship between the overall PJM score and PJM band score and academic success. When regarding the specific competencies of the competency profile, some competencies were found to display a significant relationship with academic success. From the essential competencies of the PJM, writing and reporting (M1 year), learning and researching (M1 year and final result), analysing (M1 year), and coping with pressure and setbacks (M1 year) were the only competencies that displayed a small relationship with academic success. Noteworthy, is that most of the individual PJM profile competencies that have shown a relationship with academic success included scores from both the OPQ32r and the Verify Graduate Ability Test and mostly predicted academic success for the M1 year. Only coping with pressure and setbacks as a competency that displays some statistical significance does not include scores from the Verify Graduate Ability Tests. Of the PJM competencies that were not included in the IO psychology master’s PJM competency profile, two competencies display a further statistical significance to academic success, namely creating and innovating (M1 year and final result), and formulating strategies and concepts (M1 year). Both of these competencies include scores from the Verify Graduate Ability Tests. It could then be argued that the addition of the Verify Graduate Ability Tests to the overall competency score adds validity to the competency with regard to academic success.
Taking a closer look at the IO psychology master’s PJM competency profile, it begs the question if it is useful to base access requirements to the master’s programme on an IO psychologist’s job profile. The SHL’s 20 generic competencies (SHL, 2009) were used as a point of departure and include competencies that have been found to contribute to superior performance in different roles and positions in the workplace (SHL, 2011). The profiling session should also focus on academically successful IO psychology master’s students as opposed to only focussing on the profile of a practising IO psychologist. Revisiting the IO psychology master’s student’s competency profile may also be necessary in the light of the changing world of work. This study points to some predictive competencies and measures that can be used to revise the IO psychology master’s student’s competency profile for future selection and development applications to ensure validity, reliability, credibility and fairness as required by the EEA (1998).
The empirical aim of this study was to determine whether the psychometric component of the assessment battery is a valid predictor of academic success for the IO psychology master’s students at the research institution. The cognitive component of the psychometric assessment was found to display the strongest relationship to academic success, as seen in both the Verify Graduate Ability Test sten scores and all the PJM competencies that included scores from the Verify Graduate Ability Tests.
Practical implication
The practical implication of the results from this study is that the Verify Graduate Ability Test result be used as a separate weighting in addition to the PJM scores as part of the selection process. By regarding the Verify Graduate Ability Test results separately from the PJM scores, the Verify Graduate Ability Test results will carry more weight towards the final recommendation along with access into the IO psychology master’s degree at the research institution.
Limitations and recommendations
Four areas of measure contribute towards the final acceptance of the IO psychology master’s student into the programme at the research institution. One is the academic result of the student’s honours degree, the other is the PJM score of the candidate, the third is a competency-based interview score of the student, and the last one is a simulated assignment that the student has to complete, which is independently scored. Each of these four measures contributes 25% towards the final recommendation for acceptance. This study only regarded the PJM score that resulted from the psychometric component of the selection assessment. As the sample was restricted to candidates successfully selected into the programme, a restriction of range is expected to have affected the correlations obtained, suggesting that the coefficients observed are underestimates of the actual relationships under study. The sample was further skewed by the period of study that is acceptable for a master’s degree, namely 4 years and the small number of students (only 25–30) that are selected each year for entry into the programme. It is a challenge to obtain a significant enough sample in a 5-year period, which will present students with a completed degree and, therefore, a better picture of the criterion of academic success (i.e. M1, M2 and final percentage results). As a result of the limited scope of this study, the other measures in the selection assessment have not been included in the analyses. These include the previous academic results of the student, the simulation assignment and the interview that all contribute equally to the final decision of acceptance of the IO psychology master’s student.
It is recommended that this study be replicated with a larger sample size, on the M2 and final results of the students, in order to gain results that may not be so heavily impacted by the limited sample size and current restriction of range. The previous academic results (e.g. honours results), the article assignment and the Competency Based Interview (CBI) should also be included to derive additional insight into the potential incremental validity of the full assessment battery, together with the final decision-making strategy used in line with SIOP’s (2018) recommendations. It would also be valuable to conduct a predictive validity study on the entire selection process and to consider all personality competencies of the students who have been able to complete their degrees within the 4 years, as is required, against those students who are struggling to complete the degree within the required 4 years.
Finally, it is recommended to increase the weighting of the cognitive assessment results because these measures seem to have the strongest validity coefficient in selecting IO psychology master’s students who will meet the programme requirements. The Verify Graduate Numerical Ability Test has the best predictive value for the M1, M2 and final result. Particular attention should, therefore, be given to the numerical score for selection purposes.
Selecting students for the master’s programme is driven by a dual objective. Firstly, the aim is to select students who meet the competency criteria of a practicing IO psychologist and secondly, to select students who will successfully complete the academic programme. The latter is essential to adhere to the minimum academic requirements for IO psychology registration and constituted the focus of this study. Academic success is a fundamental condition for becoming an IO psychologist; the extent to which academic success is predictive of being a good or successful IO psychologist was however not explored in this study. Selection criteria assessed through personality and simulation measures are believed to remain of value in the selection battery as they align with the IO psychologist competency profile. The need to research the extent of this value remains an unaddressed challenge.
Acknowledgements
The authors would like to acknowledge SHL for sharing the psychometric data and enabling this study.
Competing interests
The authors declare that no competing interests exist.
Authors’ contributions
The authors collaboratively designed the research project, and K.O. conducted the analysis as part of her master’s degree, with A.B. (UNISA) as primary supervisor and A.v.N. as co-supervisor. K.O. (UNISA) was responsible for data analysis, literature review and interpretation of results. A.B. and A.v.N. (UNISA) contributed significantly to the choice of literature, guided the interpretation of results and comprehensively refined the final report.
Funding information
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
Data availability
Data sharing is not applicable to this article as no new data were created or analysed in this study.
Disclaimer
The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of any affiliated agency of the authors.
References
Abiddin, N.Z. (2011). Attrition and completion issues in post-graduate studies for student development. International Review of Social Sciences and Humanities, 1(1), 15–29.
Adamiak, J., & Sauls, R.R. (2017). Influencing mathematics students’ academic success through online interventions: A case study. South African Journal of Higher Education, 31(3), 230–248. https://doi.org/10.208535/31-3-1573
Akgündüz, D., & Akınoğlu, O. (2017). The impact of blended learning and social media-supported learning on the academic success and motivation of the students in science education. Education & Science, 42(191), 69–90. https://doi.org/10.15390/EB.2017.6444
Babbie, E. (2005). The basics of social research: Research design. Thomson Wadsworth.
Barkhuizen, H., Jorgensen, L.I., & Brink, L. (2014). Exploring the role of the industrial-organisational psychologist as a counsellor. SA Journal of Industrial Psychology/SA Tydskrif vir Bedryfsielkunde, 40(1), 1–10. https://doi.org/10.4102/sajip.v40i1.1193
Bartram, D. (2005). The great eight competencies: A criterion-centric approach to validation. Journal of Applied Psychology, 90(6), 1185–1203. https://doi.org/10.1037/0021-9010.90.6.1185
Biçer, N. (2017). The influence of student-centred methods in Turkish language instruction on academic success: A meta-analysis study. Universal Journal of Educational Research, 5(4), 687–697. https://doi.org/10.13189/ujer.2017.050419
Blakeny, R., Broenen, R., Dyck, J., Frank, B., Glen, D., Johnson, D., & Mayo, C. (2002). Implications of the results of a job analysis of I-O psychologists. The Industrial-Organisational Psychologist, 39(4), 29–37. https://doi.org/10.1037/e576932011-004
Branson, H. (2014). An organisation is only as strong as its people: How to set up your purpose-driven business now. Retrieved from https://www.linkedin.com/pulse/20141204101307-369575369-an-organisation-is-only-as-strong-as-its-people/
Burke, E., Van Someren, G., Tatham, N., & Downey, K. (2013). SHL verify graduate ability test portfolio. Thames Ditton: SHL Group Ltd.
CEB (2010). Occupational Personality Questionnaire (OPQ32). Summary of Test Review. The British Psychological Society. https://ptc.bps.org.uk/test-review/occupational-personality-questionnaire-opq32
Cilliers, F., & Flotman, A. (2016). The psychological well-being manifesting among masters’ students in industrial and organisational psychology. SA Journal of Industrial Psychology, 42(1), 1–11. https://doi.org/10.4102/sajip.v42i1.1323
Cilliers, F., & Harry, N. (2012). The systems psychodynamic experience of first-year masters’ students in industrial and organisational psychology. SA Journal of Industrial Psychology, 38(2), 1–9. https://doi.org/10.4102/sajip.v38i2.992
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd edn.). Lawrence Erlbaum Associates. Hillsdale: New Jersey
Dilchert, S., Ones, D.S., & Krueger, R.F. (2019). Personality assessment for work: Legal, I-O, and clinical perspective. Industrial and Organizational Psychology, 12(2), 143–150. https://doi.org/10.1017/iop.2019.27
Employment Equity Act, No. 55. (1998). Government Gazette 400 (19370). Retrieved from https://www.labourguide.co.za/download-top/135-eepdf/file
Foxcroft, C., & Roodt, G. (2009). Introduction to psychological assessment. Cape Town: Oxford University Press.
Franklin, D. (2017). The economist: The world in 2017. New York, NY: Saratoga Springs.
Ganaie, M.U., & Haque, M.I. (2017). Talent management and value creation: A conceptual framework. Academy of Strategic Management Journal, 16(2), 1–9.
Grass, J., Strobel, A., & Strobel, A. (2017). Cognitive investments in academic success: The role of need for cognition at university. Frontiers in Psychology, 8, 790. https://doi.org/10.3389/fpsyg.2017.00790
Gratton, L. (2011). Workplace 2025 – What will it look like? Organisational Dynamics, 40(4), 246–254. https://doi.org/10.1016/j.orgdyn.2011.07.002
Health Professions Act, No. 56. (1974). Government Gazette 41100 (31433). Retrieved from https://hpcsa.co.za/Uploads/MDB/Rules%20%26%20Regulations/regulations_gnr_237_2009.pdf
Health Professions Council of South Africa. (2017). Profession board for psychology: Application for registration psychologists. Retrieved from https://www.hpcsa.co.za/Uploads/Registration_Forms/PSB/2020%20Forms/Form_24_PS_Psychologists_08072020.pdf
Jeynes, W.H. (2016). A meta-analysis: The relationship between parental involvement and African American school outcomes. Journal of Black Studies, 47(3), 195–216. https://doi.org/10.1177/0021934715623522
Joubert, T., Ilke, I., Bartram, D., Dowdeswell, K., & Lin, Y. (2015). A comparison of the psychometric properties of the forced choice and Likert scale versions of a personality instrument. International Journal of Selection and Assessment, 23(1), 92–97. https://doi.org/10.1111/ijsa.12098
Kale, E., & Etyemez, S. (2017). The moderating role of trait anxiety in the impact of self-concordance on academic success. Journal of International Social Research, 10(50), 1005–1013. https://doi.org/10.17719/jisr.2017.1729
Kappe, R., & Van der Flier, H. (2011). Predicting academic success in higher education: What’s more important than being smart? European Journal of Psychological Education, 27(4), 605–619. https://doi.org/10.1007/s10212-011-0099-9
Khan, A., Khan, S., Zia-Ul-Islam, S., & Khan, M. (2017). Communication skills of a teacher and its role in the development of the students’ academic success. Journal of Education and Practice, 8(1), 18–21.
Killen, R. (1994). Differences between students’ and lecturers’ perceptions of factors influencing students’ academic success at university. Higher Education Research and Development, 13(19), 9–212. https://doi.org/10.1080/0729436940130210
Kotzé, M., & Griessel, L. (2008). Predicting the academic performance of MBA students: A South African study. International Journal of the Humanities, 6(1), 147–156. https://doi.org/10.18848/1447-9508/CGP/v06i01/42349
Kriek, H.J. (2009). Criteria: Standards for decision-making. In P.M. Muchinsky, H.J. Kriek, & A.M.G. Schreuder (Eds.), Personnel psychology (3rd edn., pp. 45–76). Cape Town: Oxford University Press.
Lachenicht, L. (2002). Correlations. In C. Tredoux & K. Durrheim (Eds.), Numbers, hypotheses & conclusions: A programme in statistics for the social sciences (4th edn., pp. 160–180). Cape Town: University of Cape Town Press.
Landy, F., & Conte, J.M. (2016). Work in the 21st century: An introduction to industrial and organizational psychology (5th edn.). Hoboken, NJ: Wiley.
Lawler, I.E. (2011). Creating a new employment deal: Total rewards and the new workforce. Organisational Dynamics, 40(4), 302–309. https://doi.org/10.1016/j.orgdyn.2011.07.007
Machika, P.P., & Johnson, B.B. (2015). Post-graduate students’ experience of poverty and academic success at a university of technology in South Africa. South African Journal of Higher Education, 29(6), 167–181. https://doi.org/10.20853/29-6-542
Maitland, A., & Thomson, P. (2011). Future work: How businesses can adapt and thrive in the new world of work. Basingstoke: Palgrave Macmillan.
Morgan, J. (2014). A simple explanation of ‘The Internet of things’. Forbes. Retrieved from https://www.forbes.com/sites/jacobmorgan/2014/05/13/simple-explanation-internet-things-that-anyone-can-understand/#2faf68f51d09
Omarjee, L. (2015). 2025: The changing world of work, 4 June 2015. Finweek.
Önder, E., & Seyma, S. (2017). CHAID analysis to determine socio-economic variables that explain students’ academic success. Universal Journal of Educational Research, 5(4), 608–619. https://doi.org/10.13189/ujer.2017.050410
Parker, J.A., Saklofske, D.H., & Keefer, K.V. (2017). Giftedness and academic success in college and university: Why emotional intelligence matters. Gifted Education International, 33(2), 183–194. https://doi.org/10.1177/0261429416668872
Potgieter, T.E., & Van der Merwe, R.P. (2002). Assessment in the workplace: A competency-based approach. SA Journal of Industrial Psychology, 28(1), 60–66. https://doi.org/10.4102/sajip.v28i1.31
Potter, C. (2006). Programme evaluation. In M. Terre Blanche, K. Durrheim, & D. Painter (Eds.), Research in practice: Applied methods for the social sciences (2nd edn., pp. 3–17). Cape Town: University of Cape Town Press.
Puchert, J.I., Dodd, N., & Viljoen, K.L. (2017). Secondary education as a predictor of aptitude: Implications for selection in the automotive sector. SA Journal of Industrial Psychology/SA Tydskrif vir Bedryfsielkunde, 43(1), 1–13. https://doi.org/10.4102/sajip.v43i0.1416
Roksa, J., & Whitley, S.E. (2017). Fostering academic success of first-year students: Exploring the roles of motivation, race, and faculty. Journal of College Student Development, 58(3), 333–348. https://doi.org/10.1353/csd.2017.0026
Roodt, G. (2009). Validity: Basic concepts and measures. In C. Foxcroft & G. Roodt (Eds.), Introduction to psychological assessment in the South African context (pp. 55–64). Cape Town: Oxford University Press.
Salas, E., Kozlowski, S.J., & Chen, G. (2017). A century of progress in industrial and organisational psychology: Discoveries and the next century. Journal of Applied Psychology, 102(3), 589–598. https://doi.org/10.1037/apl0000206
Salend, S.J., & Whittaker, C.R. (2017). UDL: A blueprint for learning success. Educational Leadership, 74(7), 59–63.
Salkind, N.J. (2016). Exploring research, global edition. Boston, MA: Pearson.
Schmidt, C. (2006). Validity as an action concept in IO psychology. SA Journal of Industrial Psychology, 32(4), 59–67. https://doi.org/10.4102/sajip.v32i4.251
Schmidt, F.L., & Hunter, J.E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124(3), 262–274. https://doi.org/10.1037/0033-2909.124.2.262
Schreiber, B., & Yu, D. (2016). Exploring student engagement practices at a South African university: Student reliable predictor of academic performance. South African Journal of Higher Education, 30(5), 157–175. https://doi.org/10.20853/30-5-593
SHL. (1999). OPQR32 manual and user guide. London: SHL Group Ltd.
SHL. (2007). The SHL verify range of ability tests: Technical manual. Manual. Thames Ditton, London: SHL Group Ltd.
SHL. (2009). OPQ32r technical manual. Thames Ditton, London: SHL Group Ltd.
SHL. (2011). OPQR32. Thames Ditton, London: SHL Group Ltd.
SHL. (2018). Person-job match (PJM) South African research supplement. London: SHL Group Ltd.
SIOP. (2018). Principles for the validation and use of personnel selection procedures. Industrial and Organizational Psychology: Perspectives on Science and Practice, 11(Suppl 1), 2–97. https://doi.org/10.1017/iop.2018.195
Spreitzer, G.M., Cameron, L., & Garrett, L. (2017). Alternative work arrangements: Two images of the new world of work. Annual Review of Organizational Psychology and Organizational Behaviour, 4, 473–499. https://doi.org/10.1146/annurev-orgpsych-032516-113332
Taylor, C., McManus, I.C., & Davison, I. (2018). Would changing the selection process for GP trainees stem the workforce crisis? A cohort study using multiple-imputation and simulation. BMC Medical Education, 18(1), 1–81. https://doi.org/10.1186/s12909-018-1160-z
Theron, C. (2009). The diversity-validity dilemma: In search of minimum adverse impact and maximum utility. SA Journal of Industrial Psychology/SA Tydskrif vir Bedryfsielkunde, 35(1), 1–13. https://doi.org/10.4102/sajip.v35i1.765
Tinto, V. (1993). Leaving college: Rethinking the causes and cures of student attrition (2nd edn.). Chicago, IL: University of Chicago.
Tsingos-Lucas, C., Bosnic-Anticevich, S., Schneider, C.R., & Smith, L. (2017). Using reflective writing as a predictor of academic success in different assessment formats. American Journal of Pharmaceutical Education, 81(1), 1–8. https://doi.org/10.5688/ajpe8118
Van Aarde, N., Meiring, D., & Wiernik, B.M. (2017). The validity of the big five personality traits for job performance: Meta-analyses of South African studies. International Journal of Selection and Assessment, 25(3), 223–239. https://doi.org/10.1111/ijsa.12175
Van Vuuren, L.J. (2010). Industrial psychology: Goodness of fit? Fit for goodness? SA Journal of Industrial Psychology/SA Tydskrif vir Bedryfsielkunde, 36(2), 1–16. https://doi.org/10.4102/sajip.v36i2.939
Viviers, R., & Van Niekerk, A. (2012). Assessment and development towards graduateness and employability of masters’ students in industrial and organisational psychology. In M. Coetzee, J. Botha, N. Eccles, H. Nienaber, & N. Holtzhausen (Eds.), Developing student graduateness and employability: Issues, provocations, theory and practical guidelines (pp. 263–276). Randburg: Knowledge Resources.
Wilcox, M. (2016). Effective talent management: Aligning strategy, people and performance. Abingdon, Oxon: Routledge.
Xu, M., David, J.M., & Kim, S.H. (2018). The fourth industrial revolution: Opportunities and challenges. International Journal of Financial Research, 9(2), 90–95. https://doi.org/10.5430/ijfr.v9n2p90
|
Email this article 
Hide
Show all
