Original Research

Investigating the data science talent gap: Data practitioners’ perspectives

Wilma Coetzee, Roelien Goede

Received: 01 Feb. 2025; Accepted: 18 Mar. 2025; Published: 29 Apr. 2025

Copyright: © 2025. The Author(s). Licensee: AOSIS.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Orientation: Data professionals are indispensable assets for companies, offering crucial insights that provide a competitive edge in economically demanding times.

Research purpose: The purpose of the study is to gain insight into the skills gap of South African data practitioners, explore job-related stressors and identify factors leading to job dissatisfaction.

Motivation for the study: Universities need to identify the skills their graduates are missing to adjust their data science programmes accordingly. To prevent the loss of skilled data professionals, businesses must understand the challenges these employees face and find ways to better support them.

Research approach/design and method: The study adopted a mixed-method approach, using online surveys featuring both closed and open-ended questions, with 46 respondents, data professionals mainly working in banking, insurance and consulting.

Main findings: The surveys aimed to determine what these professionals view as skills gaps, and what their job experiences, stressors and coping mechanisms are.

Practical/managerial implications: The results highlight the importance of using real-life data to empower students to practise data pre-processing. Students should be challenged to engage in data visualisation, report-writing and presentations. Universities and employers should implement soft-skill development programmes, incorporating self-care for mental health. Closer collaboration between universities and industries is needed.

Contribution/value-add: This study identifies critical skills gaps among South African data professionals and explores workplace stressors that affect employee retention and job satisfaction. By bridging academia and industry expectations, the study provides actionable insights for curriculum design and human resources strategies to support data practitioners’ well-being and professional development.

Keywords: data scientist; employability; job retention; job satisfaction; soft skills; South Africa; talent gap.

Introduction

Despite the increasing demand for data professionals, South African graduates often face skill gaps that hinder employability. Additionally, workplace stressors, including high-pressure deadlines and technostress, might contribute to job dissatisfaction and turnover. This study aims to identify these gaps and challenges, offering insights to improve human resource management (HRM) practices and academic curricula.

Professionals such as data engineers, data scientists and business analysts are in high demand to unlock business insights through cloud computing, big data analytics and machine learning (Aujla et al., 2022) Companies are keen to hire them to gain a competitive edge over their rivals (Mikalef & Krogstie, 2019). Industries worldwide are seeking these so-called ‘unicorn data scientists’ – experts in mathematics, statistics, computer science, artificial intelligence and many more – that ‘rarest of breeds that can do it all’ (Baškarada & Koronios, 2017, p. 65). However, the shortages in providing the talent to do this are intensifying worldwide: the U.S. Bureau of Labor Statistics projected a 28% annual growth in jobs requiring data science expertise by 2026 (Janssen, 2022).

Graduate data professionals need a wide range of competencies and workplace skills, yet most university curricula worldwide ignore many of those required in both research and industry (Demchenko et al., 2021). According to Price Waterhouse Coopers (2019), the skills gap is a major obstacle for organisations aiming to leverage the potential of digital technologies, particularly in Africa. For this reason, it is essential to revamp traditional educational courses to reflect the multidisciplinary nature of data science (Demchenko, 2017), also in South Africa (Msweli et al., 2023).

Universities should not only focus on the skills gap but also prepare students to cope with stress. Data professionals deal with enormous amount of data of various kinds and must master the latest technologies, which exposes them to technostress (Kot, 2022). This kind of occupational hazard is a problem for both employees and their employers. Stressful working conditions can affect employee well-being, physical health, work performance, job satisfaction and engagement (Aktan & Toraman, 2022). Although modern technologies have made it possible for employees to work flexible hours, allowing them to balance work and personal life better, these same technologies have also brought negative consequences, such as the pressure to be constantly available, which can lead to stress and burnout (Ninaus et al., 2015). Therefore, these professionals also need soft skills such as self-regulation and emotional intelligence to cope with stresses at work as well as the ability to communicate, to show leadership, be creative, pay attention to detail and be a team player (Smaldone et al., 2022).

Studies worldwide have identified the skills needed in the commercial sector, primarily using online job search data (Fang & Jiyuan, 2024; Smaldone et al., 2022; Stanton & Stanton, 2023).

However, according to Mikalef and Krogstie (2019), little research is available on the importance of soft and interpersonal skills for data professionals and what managers in this field need to be proficient.

Limited studies have been conducted in South Africa on the skills needed by the local industry and the prevailing skills gaps and work experiences among young data professionals because data science is still a relatively new occupation (Goretzki et al., 2023). Even globally, there is a shortage of research on the job stressors faced by data professionals, their coping mechanisms, how universities can best prepare students for the challenges of this environment and how employers can best support them. This study addresses this research gap.

The mixed-method study reported here delves deeper into the skills gap, job-related stressors and factors contributing to job dissatisfaction, as revealed by South African data professionals. Recommendations are made based on the insights gained. Universities can use the information to improve their curricula to provide their data analytics students with the required skills. Companies can use this information to develop strategies to retain data professionals and look after their well-being.

Key concepts are explained in the next section. Thereafter, the results of the analysis of an online survey of 46 data professionals are presented, followed by a discussion of the implications and recommendations on how educational practices can be improved and how companies can provide better support to their data professionals.

Key concepts

Employability

Employability can be defined as ‘the ability to realise potential through sustainable employment’ (Hillage & Pollard, 1998, p. 24), implying that a person should be able to find employment based on the potential they have and keep the employment because of realising that potential (to the benefit of the employer).

For our purpose, we will focus on the data professional as ‘the person’ and imply that the term encompasses all of the following, as proposed by the EDISON Data Science Framework (EDSF) (Demchenko et al., 2021):

• Managers – chief data officer, data science manager, data infrastructure manager.
• Professionals – data scientist, data analyst, data engineer, business analyst.
• Database professionals – database designers and administrators.

‘Finding employment’ is easier when a skills gap can be averted and by aligning academia with employers’ needs (Smaldone et al., 2022). ‘The potential’ of a graduate can be seen in previous accomplishments such as degrees, grades, achievements and experience. ‘Keeping employment’ implies that the employee meets the demands of the employer. This depends on the employee’s ability to use their acquired skills and develop new ones.

Employers seek employees who possess the essential skills, both technical and soft, so they can effectively fulfil their job responsibilities. According to Nickson et al. (2012), soft skills greatly affect employability and career progression. Employees need strong soft skills to boost employability and thrive in the workforce (Seetha, 2014). Employees tend to overestimate their soft skills; they often believe they possess them at a higher level than their employers perceive (Tsirkas et al., 2020).

Adhvaryu et al. (2018) investigated the feasibility of enhancing employees’ soft skills, the effects of improved soft skills on workplace outcomes and whether it is beneficial for companies to offer on-the-job soft-skills training to their employees. They found that those who received such training were 20% more productive than the control group after the programme. Within 8 months of completing the training course, the net return to the firm increased by more than 100%.

Job satisfaction and turnover intention

Job satisfaction is the feeling of fulfilment and enjoyment derived from work, a strong bond with co-workers, autonomy, sufficient resources and overall life quality (Montuori et al., 2022; O’Leary et al., 2009). High job satisfaction can positively impact employee health, performance, energy, dynamism, inspiration and commitment to work (O’Leary et al., 2009; Phuong & Vinh, 2021).

Herzberg’s two-factor theory of motivation distinguishes between motivators (intrinsic factors that encourage employees to work harder, contributing to job satisfaction), and hygiene factors (extrinsic aspects that do not necessarily lead to job satisfaction but can cause dissatisfaction if they are inadequate or absent) (Nickerson, 2023). Job dissatisfaction can lead to work absenteeism, job turnover and even burnout (Montuori et al., 2022). Turnover intention can be defined as ‘the conscious and deliberate wilfulness to leave the organisation’ (Tett & Meyer, 1993, p. 262). Job satisfaction significantly predicts turnover intention: the less job satisfaction workers experience, the more likely they are to leave the company (Oosthuizen et al., 2016).

A study on the retention of professionals in technology-based organisations found that such people are likely to leave their job for reasons such as lack of job satisfaction, limited opportunities for growth, absence of a clear path for advancement, better working conditions, inadequate management practices and individual career planning (Steil et al., 2022).

Companies should invest in the job satisfaction of their employees who, when satisfied, experience both physical and psychological well-being, are more productive and less likely to leave the company, whereas those dissatisfied tend to show absenteeism, psychological withdrawal and eventually seek another job (Rothmann & Coetzer, 2002).

Factors affecting job satisfaction and turnover intention

Managing the overwhelming volume of diverse data at work, processing and using it effectively, causes psychological stress (Kot, 2022). This is called technostress, which employees experience because of various factors such as multitasking, information overload, constant contact via electronic devices, frequent system updates and the need to constantly stay up-to-date with the latest technological advancements (Ragu-Nathan et al., 2008). Technostress may adversely affect employee satisfaction (Kot, 2022).

Aktan and Toraman (2022) claim that stressful working conditions affect employee well-being, physical health, work performance, job satisfaction and job engagement among employees. Companies can help to alleviate technostress in their workforce by providing social and technical support, digital literacy facilitation and involvement whereby workers are consulted and called in when technology changes are implemented (Hwang & Cha, 2018; Stana & Nicolajsen, 2021; Tarafdar et al., 2017).

Oosthuizen et al. (2016) suggest that managers and human resource practitioners should introduce work–life balance initiatives to reduce IT workers’ turnover intentions and improve their job satisfaction. Work–life balance is the level of engagement and satisfaction people experience in their career, family and non-work activities and is crucial for a fulfilling lifestyle (Greenhaus et al., 2003). Soft skills may assist in enhancing job satisfaction, raising awareness of the importance of self-care and improving work–life balance.

Research design

The research method followed is discussed further in the text.

Study design

As actionable insights were sought, it was decided to combine the numerical precision of quantitative methods with the narrative richness of qualitative data to help paint a fuller picture. A mixed-method approach was followed, in which a positivistic paradigm was used for the Likert-scale quantitative data and an interpretive paradigm for qualitative data. This approach strengthens trustworthiness through triangulation of the data as the combination of quantitative data and qualitative commentary helps to cross-verify the results. A factor analysis of the quantitative data is performed to check if the Likert scale measures are reliable and valid.

Thorne (2013) suggests that researchers who engage in anti-oppressive research may find interpretive description appealing due to its focus on promoting change because it aims to generate action-orientated knowledge by translating respondents’ voices into actionable insights to bring about change. As we studied the personal experiences and perspectives of data professionals to make recommendations for improving universities’ courses for data analytics students, we deemed interpretive description suitable for addressing the qualitative part of this study.

Objectives

The study had three main objectives, each with sub-objectives:

1. Gain insight into the skills gap:
   • 1a. Identify the skills required by industry.
   • 1b. Reveal skills that are lacking among young data practitioners.
   • 1c. Explore how universities can better prepare students for the workplace.
2. Explore job-related stress of data practitioners:
   • 2a. Identify job stressors.
   • 2b. Discover mechanisms they use to cope with stress.
3. Determine how data practitioners experience their jobs to identify factors that could potentially lead to job dissatisfaction:
   • 3a. Identify their job attributes, for example, challenging, stressful, among others.
   • 3b. Discover the aspects they like and dislike about their jobs.

Measuring instrument

An online survey was created, consisting of questions on the demography of the respondents, eight open-ended questions and a section with 50 closed questions.

The survey included open-ended questions that covered topics like preferences, grievances, sources of stress and ways of dealing with them in the work environment. Respondents were also asked to share whether their education had sufficiently equipped them for their careers. Additionally, they were invited to advise new graduates on managing work-related challenges, interpersonal dynamics and mental well-being in the workplace.

The closed questions were centred on topics that arose in an exploratory study analysing online South African job postings for data professionals and recurrent themes noted in the literature (Coetzee & Goede, 2024; Smaldone et al., 2022; Stanton & Stanton, 2023). These questions used Likert scales. Face validity was tested by performing a small pilot study to identify any gaps or issues with the questions.

Thirty-one questions were asked to ascertain which skills young data practitioners typically lack, where 1 = ‘Generally, don’t lack’ and 4 = ‘Almost always lack’. Exploratory factor analysis (EFA) with varimax rotation yielded five factors for the skills lacking with the majority of Cronbach’s α values above 0.75. The total variance explained was 47.3%.

Nineteen questions measured if the respondents experience their job a certain way, where 1 = ‘Never’ and 5 = ‘Almost always’. Here, EFA with varimax rotation yielded five factors. The Cronbach’s α values exceeded 0.60, and the total variance explained was 57.6%. The quantitative section of the instrument was, therefore, deemed as fairly reliable. The analyses are shown in the Results section.

Data collection

Purposive sampling was employed to ensure that data professionals from different industries, roles and levels of experience were included. The study population represented the South African banking, insurance, consultation, business and agricultural sectors. This group varied from those who had recently entered the labour market to seasoned and highly experienced staff. Some are working as technical experts, whereas others are involved in management.

An explanation of the research and a link to the online survey was sent via email to 118 data professionals. This approach ensured voluntary anonymous participation. Survey data collection took place from 08 August 2023 to 29 August 2023. Semi-structured online interviews were conducted with two respondents, who offered to elaborate on their survey responses in a follow-up interview. The interviews were recorded and transcribed using MS Teams’ utilities.

Sample

A total of 46 responses were received. The demographics of the study population in terms of the industry, qualification, number of years worked and level in their company illustrate the broad representation of the sample (Figure 1).

FIGURE 1: Qualification, industry, job category and work duration of respondents.

The respondents represent graduates with 3-year bachelor’s, honours (4th year) and master’s degrees. As most graduates from the authors’ host university are employed in banking, we were particularly interested in the views of data practitioners in this sector. The banking sector represented 65.2% of the sample, the insurance sector 8.7%, consultation 13%, and business 4.3%. Agriculture, mining, telecommunication and financial marketing each represented 2.2% of the sample. Most respondents held advanced technical positions (56.5%), although entry-level (10.9%) and management positions (32.6%) were also well-represented. The majority had been working between 6 and 10 years (54.3%), but less experienced employees (41.4%) were also included in the sample.

Capturing and storing

All the data were captured on an Excel spreadsheet and stored in a secure drive. The qualitative data were extracted to a separate spreadsheet and imported into ATLAS.ti for further analysis, whereas the quantitative data were imported into R for explorative data visualisation and calculation of descriptive statistics.

Data analysis

For the answers containing Likert-scale values, statistics of central location were calculated and ranked to indicate the issues that need to be addressed (according to the respondents).

Qualitative data were coded, and emerging themes were identified. Table 1 provides an example of the coding of a response regarding stressors at the workplace. It also shows themes that emerged as possible causes of stress and those that emerged as coping mechanisms.

Recurring themes quickly emerged and rich data were obtained where multiple examples could support each theme. The researchers are satisfied that sufficient data saturation was achieved.

Ethical considerations

The Faculty Ethics Board of the authors’ host university granted ethical consent for the study on 23 February 2023 (reference number NWU-01210-23-A9). All respondents of the survey provided written informed consent. The interviewees provided verbal consent (which was recorded) before the interviews. The online surveys were anonymous, and no personally identifiable information about the respondents was collected, who completed the survey voluntarily.

Results

Objective 1: The skills gap

When asked if their tertiary education prepared them adequately for their current position, a respondent indicated that there is a gap between what universities offer and what is needed in the industry:

‘No, and that’s mainly down to the course I selected. The course I selected was heavily focused on mathematics and statistics fundamentals and not on the programming and software development. This meant that we studied some of the ideas behind functions that we would later use in the real world instead of applying the tools that already have the formulas behind them. Yes, it is valuable to understand where the tools you’re using come from, how they are constructed and on which principles they are based, but in the professional world, what matters at first is applying the latest tools to problems.’ (Respondent 8, Senior Consultant, Consulting Industry)

Objective 1a: Skills required by industry

Next, we outline the different skills the respondents identified as important, and which they indicated junior data practitioners typically lack. We also share their views on how universities could enhance their collaboration with industry.

Software and technical skills: Universities should ensure that their students are familiar with the most commonly used software packages in their industry. The study revealed that the top ten software skills, from most to least in demand, are as follows: Excel, PowerPoint, Word, SQL, PowerBI, SAS Python, R, Tableau and SAP.

When asked to recommend which technical skills students should acquire, the respondents advised that they practise coding, learn to work on the cloud, use big data and use a GIT repository. Apart from learning Excel, PowerPoint and Microsoft Word, the students should acquire expertise in SQL, PowerBI, SAS and Python. They stressed the importance of students working on real-life data. Interviewee 1 emphasised the significance of proper data extraction and cleaning to derive accurate conclusions, something that can be learnt only by working with messy real-life data. He highlighted the importance of mathematical studies to facilitate logical thinking. Students should be challenged to write reports, visualise data and present their results. Universities should raise awareness of the different career paths in data that students can pursue:

‘Engage with businesses in practice to: land small practice projects for students; land larger projects or on-the-job training as part of an honours or master’s degree; keep up to date with technology in use. Organise “take a friend to work” days/weeks with businesses for students to see the real day-to-day demands of a business.’ (Respondent 29, Manager, Banking Industry)

The significance of students being able to apply their acquired knowledge was emphasised in the below quotation:

‘While analytical positions in South Africa [insurance, banking, medical, pharmaceutical, psychology, etc.] are not devoid of using theoretical statistics, the greater and more general need is for students to be able to apply statistical principles practically in their work.’ (Interviewee 2, Manager, Banking Industry)

Soft skills: Students should acquire soft skills, such as learning how to secure a job, how to write a curriculum vitae (CV) and present themselves at an interview. Important soft skills also involve developing adaptability, embracing change, self-assessment, reflection and devising improvement strategies. Students should be aware also of the significance of having a support system, learning to say ‘no’, taking breaks, adopting healthy habits such as eating well, regular exercise, relaxation, spending time with friends and family, caring for their mental health and managing stress:

‘Old habits die hard. Sometimes as a student, you want to cover all of the material at all costs. This means that you acquire terrible habits. Bad eating habits to avoid breaking out of focus from your study sessions; not taking breaks, skipping out on sleep; not socialising to maximise time spent studying; not exercising to maximise time spent on studying. These are all “sustainable” bad habits in a structured environment where recess exists and walking from class to class as a form of exercise, scheduled classes as a hidden way to force breaks in between sessions. But when you get to the professional world, no one will schedule breaks and leave days for you. Avoiding colleagues to maximise productivity isn’t the answer. So above passing and distinctions, students need to be encouraged to adopt a routine, socialise in a healthy and responsible way and build the right habits in time for the professional world. Burnout, if you’re unlucky [sic], can have a long-lasting impact. Help students to build the habits needed to avoid it in their future. And encourage students to reach out for help. Furthermore, reach out to students. People who need help the most are often passive.’ (Respondent 8, Senior Consultant, Consulting Industry)

People skills: People skills are critical for success in the workplace. Students need to know how to be effective colleagues, to act professionally, to manage people, to network effectively, to manage stakeholders and their expectations, to collaborate in a cross-disciplinary way, to conduct meetings and also how to communicate effectively. They need to know about the importance of a mentor, the power of effective teamwork and how to be a reliable team player:

‘Stimulate them to think about social cohesion and thinking about the broader team and not so much their individual goals. When you are conscious of the needs/desires of others in the workplace your individual goals are also easier to negotiate.’ (Respondent 6, Risk Analyst, Banking Sector)

Objective 1b: Perceived lack of skills among younger colleagues

The study population was asked whether they think their younger colleagues lack certain skills, based on a 4-point Likert scale: 1 = ‘Generally, don’t lack’, 4 = ‘Almost always lack’.

A factor analysis performed on 31 items yielded the results presented in Table 2. The factors identified include self-regulating skills, intra- and interpersonal skills, technical skills, management skills and life-outside-of-work skills.

Mean values were calculated for each of the 31 items. High mean values indicate that most of the younger colleagues were perceived to lack these skills (Table 3). Among the top 10 skills they lack, four belong to the ‘management’ factor, four relate to the ‘self-regulating’ factor and two illustrate the ‘intra- and interpersonal’ factor.

These results indicate that young data practitioners’ presentation and report-writing skills, ability to network and business insight are considered not up to standard. They fall into the trap of trying to make something perfect rather than delivering on time. They typically do not cope well with stress, struggle to prioritise, labour to manage their time and do not take ownership. They have difficulty thinking ‘out of the box’.

Objective 1c: Universities’ role in closing the gap

Many respondents emphasised the importance of closer relations between educational institutions and the commercial world. They suggested different ways to achieve this, such as by industry visits, where students can learn about employment opportunities, conducting ‘take-a-friend-to-work’ days, where students can visit businesses and observe what the day-to-day job looks like, and offering vacation work. However, interviewee 1 pointed out that it is not easy to find small enough tasks to give to students as vacation work. He proposed that analysts could share their stories with students in a ‘day in the life of an analyst’ manner, where they talk about their assignments, the challenges they have faced, and how they solved the problems.

Objective 2: Job-related stress of data practitioners

Respondents were asked to comment and reflect on aspects of their jobs that create stress, and how to cope with them.

Objective 2a: Job stressors

The majority of those surveyed indicated that tight deadlines were the primary cause of stress, as well as high pressure and a heavy workload.

Other stressors mentioned included dealing with multiple stakeholders, factors beyond one’s control, unrealistic expectations, lack of communication and self-regulation, technical challenges, work relationships, poor planning, and shortage of skilled staff:

‘What causes work stress is having projects at short notice, being given less time to work on them [tight deadlines], which results in one ending up working overtime with no plans of working overtime. At times, you just have to pause everything [personal life, school, hobbies] and just focus on work. What is also stressful the most is that you barely can plan ahead and most things are always out of one’s control.’ (Respondent 28, Consultant, Banking Industry)

‘High demands, lots of responsibility in managing people in a very technical environment and the financial pressures in terms of reducing our cost ratios.’ (Respondent 3, Manager, Insurance Industry)

‘The ever-changing nature of consulting in the tech space can be stressful. I often need to quickly switch between various technologies and industries, and often, this is overwhelming. The pressure to meet tight deadlines and navigate intricate client requirements can contribute to stress as well. Also, the challenge of aligning technical recommendations with client preferences or constraints can be very frustrating. When you add the multiple ways relationships with clients and colleagues can go wrong and sometimes does go wrong, you get stressed.’ (Respondent 8, Senior Consultant, Consulting Industry)

Objective 2b: Coping mechanisms

Communication was widely considered to be important to cope with stress as are project management skills, and being able to set boundaries. Self-regulating abilities, self-care and time management were also mentioned frequently. Some respondents reported that they turned to unhealthy ways of dealing with the stress, such as procrastination, or taking to alcohol:

‘In order to cope with such stress, it is important to write down a well thought out plan with the prior identification of any possible hindrances. More frequent communication is important to avoid any wasted efforts.’ (Respondent 15, Manager, Banking Industry)

‘I always ask questions such as why we are doing this and what are the benefits to company and clients as well. Never sit with a problem for too long, engage with team members, don’t be shy to ask for their help.’ (Respondent 24, Senior Data Scientist, Banking Industry)

‘I have learned to communicate my frustration in a way which would not come across as complaints. I tend to work on big projects and therefore I fear that they might fail. In order to calm down I try to integrate the business into my processes so that everyone is aware of the details and ensured that all parties are vested in the success of the project. So basically, I cope with work stress by being more self-aware and communicate concisely all the time.’ (Respondent 26, Senior Data Scientist, Banking Industry)

‘Well, I try to prioritise effective time management. Planning and organizing my tasks allow me to allocate time appropriately and ensure that deadlines are manageable. I’ve also found that open communication with team members and clients is helps. Discussing project progress, challenges, and potential adjustments helps manage expectations and prevents misunderstandings. Taking breaks is another key aspect of my stress management approach. Stepping away from work, even for a short while, helps clear my mind and regain focus. Engaging in activities I enjoy outside of work, such as hobbies or spending time with loved ones, also helps. Exercise and meditation also go a long way. And if all else fails, leaning on my belief in God often does the trick.’ (Respondent 8, Senior Consultant, Consulting Industry)

‘I cope by making sure I switch by going to the gym and trying by all means to not work over weekends.’ (Respondent 13, Senior Data Scientist, Business Sector)

‘My stress is caused by hard-to-achieve deadlines that end up requiring me to work longer hours to achieve. The lack of empathy and understanding from my line manager adds to the stress and I’m currently not coping well with the stress. I’ve resorted to drinking and sleeping pills.’ (Respondent 38, Senior Quantitative Analyst, Banking Industry)

(Interviewee 1, Manager, Banking Industry) recommended that employees set realistic goals and communicate them with their line manager. He mentioned that prioritising important tasks can alleviate stress and avoid over-commitment.

A Sankey graph was created using ATLAS.ti to see which coping strategies are associated with which stressors (Figure 2). The codes of the open-ended questions asking about the causes of respondents’ stress and how they typically cope with stress, were portrayed visually to see which codes co-appeared in a respondent’s answers. The graph can visualise complex relationships. Each colour represents a different strategy data professionals use to cope with particular stressors. The thicker the line linking the strategy and the stressor, the more these two themes co-occurred. We see that project management, time management, and communication all co-occur with the four principal stressors. Communication was highly associated with deadlines and pressure. Interestingly, project management was deemed more important than time management. Setting boundaries, regulating oneself, and practising self-care also emerged as useful strategies for handling different stressors. Unhealthy coping strategies co-occurred the most with ‘deadlines’.

FIGURE 2: Sankey diagram indicating co-occurrences of codes relating to coping strategies and stressors.

Therefore, universities should ensure they teach students principles of project management and time management, how to effectively communicate, set boundaries, regulate themselves and practise self-care.

Objective 3: Job satisfaction: Nature of job, likes and dislikes

We also explored how respondents perceive their roles as employees, and what they like and dislike about their roles.

Objective 3a: Nature of the job

A five-point Likert scale was used to record how respondents experienced certain aspects of their job, where 1 = ‘Never’ and 5 = ‘Almost always’. A factor analysis was performed on these 19 items (Table 4). Five factors were identified: stimulating, stressful, working conditions, support, and vacation. The data practitioners who responded typically view their job as stimulating but stressful. The working conditions are mostly positive. Good remuneration, flexible working hours, the ability to work remotely, and sufficient vacation make up for the deadline-driven environment.

Median values were calculated for each of these items, sorted from large to small (Table 5). High median values (close to 5) reflect how most respondents experience their jobs, whereas low values (close to 1) illustrate how relatively little they do.

Table 5 illustrates that most respondents have deadline-driven jobs that are challenging, stressful, and require much overtime. Working overtime regularly results in prolonged exposure to electronic devices, leading to increased levels of technostress. This condition may lead to job dissatisfaction and ultimately to resignation. These respondents are often expected to keep up with new technical developments and most do not find it difficult to learn new things as required. Many factors typically associated with high job satisfaction were noted. Most respondents find their jobs interesting and enjoyable, and many are allowed flexible working hours and remote work. They mostly experience that their managers understand the complexity of their assignments and are quite supportive. Overall, they are satisfied with their remuneration and the number of leave days they receive.

Very few regret having chosen their career. Junior staff often do repetitive work, whereas senior technical personnel and managers do not find their work repetitive or boring. The latter typically have higher stress levels and more deadline-driven jobs than junior employees.

Objective 3b: Likes and dislikes of the job

The responses of the study population who were asked what they liked and disliked about their job were coded using ATLAS.ti to identify themes. Most comments centred on intrinsic job satisfaction factors, such as being mentally stimulating and impactful, personal growth, autonomy and collaboration opportunities. Achieving a good work–life balance was also noted. One respondent stated that the pay was good:

‘I find my job fun and enjoyable because I can find a sense of satisfaction from immersing myself in intricate problem-solving challenges and my job gives me the opportunity to engage in meaningful work that has a measurable impact. My job empowers me to break down challenges into their component parts and use analytical skills and innovative problem-solving to deliver a solution. This really resonates with me, because that’s exactly what I have been doing for years in my spare time, unravelling intricate issues and meticulously addressing them, especially in tech.’ (Respondent 8, Senior Consultant, Consulting Industry)

‘Every day is different and I am exposed to every department in the business. There is not a person in the company that I don’t work with at some stage.’ (Respondent 45, Business Analyst, Agricultural Industry)

The fact that their job is stressful, ranked highly among the dislikes, followed by work–life balance being a struggle. Several respondents indicated that they have a high workload and tight deadlines and that a lack of capacity in their company contributes to this:

‘Projects are seasonal. I am a highly neurotic person so the off-season and uptick in projects causes a bit of chaos at the beginning and the end of a project. I thrive on regularity and consistency. The pressure to meet tight deadlines and deliver results within constrained timelines, sometimes impossible timelines, can lead to stress and burnout. The need to rapidly adapt to new client requirements and expectations, while simultaneously managing project complexities, can create an environment where work-life balance becomes a struggle. As a consultant, I often encounter situations where technical recommendations do not align with the client’s preferences or business constraints. This is frustrating because you want to implement solutions that adhere to best practices and standards. But if you are compelled to compromise by the client and leadership, there’s not much you can do. You have to comply and deal with the consequences which always show up in a system at a later point as a bug. Unfruitful meetings and passive colleagues can be annoying. Not having the time for self-improvement and learning is also not the best.’ (Respondent 8, Senior Consultant, Consulting Industry)

‘I don’t have work-life balance. I’m always stressed and anxious. I seldom have free time to rest or eat lunch.’ (Respondent 38, Senior Quantitative Analyst, Banking Industry)

‘Stressful. Reporting to the regulators means there is zero tolerance for errors. A lot of production changes involved to keep up with rule changes, adds to the pressure.’ (Respondent 40, Data Analyst, Banking Industry)

Some respondents saw work–life balance as a ‘like’, whereas others indicated the opposite. This may be because of personal coping strategies, the employee’s position, their managers or their company.

Incompetent management was also indicated as something disliked:

‘When bad calls are made by top management because they don’t understand intricacies fully and/or refuse to accept that there are certain outcomes that are beyond our control. This results in unrealistic expectations in an environment that’s already under severe pressure.’ (Respondent 46, Manager, Banking Industry)

Whereas some respondents indicated that they like their job because it is challenging, others find it too challenging and dislike it:

‘It is too demanding, I have to constantly do research and come up with new solutions to address today’s problems.’ (Respondent 2, Data Scientist, Banking Industry)

Various aspects of the job were mentioned, such as administration, reporting, documentation and meetings, which are perceived as boring and mundane. Some indicated that having to rely on other people to be able to do their jobs can be frustrating as it often leads to deadlines being missed. (Respondent 29, Manager, Banking Industry) voiced some dislikes that relate to different aspects of the job:

‘Knowing that much of the effort is futile due to corporate politics, governance processes, misguided focus, system limitations.’

Discussion

The study involved exploring the perspectives of 46 data practitioners on the skills gap in data science, the stressors affecting data professionals and potential causes of job dissatisfaction. The researchers used mixed methods to gain a comprehensive understanding of the data, the findings from which are expected to contribute significantly to the understanding of the complexities of data analytics education and the retention of data professionals.

We focused on the skills gap between what employers need and their experience of the skillset of young graduates entering the workforce. Our survey revealed that South African universities are fairly successful in preparing students with technical skills such as data analysis, modelling, programming and applying popular software. However, some areas need improvement. Consequently, we recommend that universities address the following in data science curricula: presentation skills, networking skills, business acumen, stress management, prioritising skills, time management, taking ownership, report writing and innovative thinking.

These findings reflect researchers who found that apart from hard skills, a wide spectrum of soft skills such as self-regulation, teamwork, critical thinking, communication and having business acumen, is essential for data professionals (Rawboon et al., 2021; Smaldone et al., 2022; Stanton & Stanton, 2019, 2020).

Our study population suggested fostering stronger connections between higher education and the industry, as well as providing students with more exposure to the world of data practitioners. This reflects the findings of Delavande et al. (2020), who advocated the need for a synergetic relationship between higher education and the commercial sector, and Smaldone et al. (2022), who encouraged an active multi-stakeholder approach that oversees the skills needed for data professionals to ensure their employability. Our respondents recommended that:

• Universities could organise industry visits for students.
• Companies could provide vacation work opportunities for students.
• Data practitioners could share their stories: ‘a day in the life of an analyst’.

Moreover, we explored the stressors data practitioners experience. Tight deadlines and much pressure and responsibilities were the predominant stressors mentioned. Other factors were the multiple stakeholders that had to be satisfied, conditions beyond one’s control, insufficient communication, lack of planning, insufficient skilled staff and poor self-regulation.

Our respondents also elaborated on effective coping mechanisms to handle stress, which must be taught to students. They include honest and open communication, project management, setting achievable goals, establishing personal boundaries, self-discipline, self-care and time management.

Communication is important because learning to speak the managers’ language helps data scientists communicate better with them and respond effectively to their expectations (Goretzki et al., 2023). Almgerbi et al. (2022) highlight the importance of project management skills, Mikalef and Krogstie (2019) explain the importance of goals and prioritising, whereas Caeiro-Rodriguez et al. (2021) address the significance of self-discipline and time management.

Most of the data professionals in the study revealed that they experience job satisfaction in aspects of their work that align with Herzberg’s motivators. They experience a strong sense of accomplishment when they successfully build an accurate predictive model or solve complex data challenges. Knowing that their work is meaningful and that they can take ownership of projects contributes to their job satisfaction. However, as Goretzki et al. (2023) explain, data practitioners often struggle with the ambiguity of their job – on the one hand, it is sophisticated and challenging (e.g., the modelling aspect), but it is also mundane (e.g., collecting, preparing, and cleaning the data). Performing hours of mundane tasks can be frustrating, and lead to job dissatisfaction.

While the presence of Herzberg’s hygiene factors does not guarantee job satisfaction, its absence can cause job dissatisfaction. Generally, data professionals are not dissatisfied with their salaries, job security or general working conditions such as being able to work remotely or flexible working hours. These are not major causes for job dissatisfaction. Extrinsic factors contributing the most to job dissatisfaction include working under tight deadlines, experiencing high levels of stress, working overtime, dealing with technostress, having to depend on other team members, facing frustrating delays and being subject to regulatory scrutiny, which leads to long implementation delays.

Coping effectively with these stressors is important for employees’ work–life balance, job satisfaction and mental health. A company’s success relies heavily on a small group of highly skilled and knowledgeable workers. If they choose to leave the business, the business’s competitive advantage can be put at risk (Ramsey & Barkhuizen, 2011). This is of particular concern in the IT sector where talented professionals have many career opportunities globally (Oosthuizen et al., 2016). This study can assist companies in creating effective retention strategies to retain their data professionals.

Employers should also be aware of the stressors data professionals face and find ways to reduce them, as these may lead to resignations (Rothmann & Coetzer, 2002). Looking after the mental well-being of their employees should be a high priority. We propose that companies should:

• Seek ways to create more flexible deadlines, manage unrealistic expectations and ensure effective planning.
• Create channels where their employees feel safe to communicate if they are struggling.
• Provide ongoing training on how to cope with stress, how to practise self-care, communicate, be a good team player and create a healthy work–life balance.

Limitations

This study addressed only data professionals who work in South Africa. Although care was taken to include staff from a wide variety of industries and positions, the majority of respondents were working in banking. The sample size was fairly small, making generalisation of the results difficult.

Future research and recommendations

Additional surveys and interviews are required to obtain a broader picture of the experiences and insights of data professionals, particularly those working in other organisations and countries. Research should be conducted on how to effectively teach soft skills and stress-coping mechanisms to data analytics students.

Conclusions

A mixed-method study was conducted to identify gaps in the education of South African data professionals and explore their job stressors and causes for job dissatisfaction.

It became evident that professionals in the field think there is a gap between academia and the industry that calls for closer collaboration. Students should be exposed to real-world ‘dirty’ data and understand the business world better. Skills needed to flourish in industry should be developed, like writing and presentation skills, networking, critical thinking, teamwork and self-regulation. Teaching students healthy stress-coping skills and the importance of self-care is essential.

Data professionals are required to constantly work in a highly competitive, technologically advanced environment characterised by tight deadlines, technostress and frustrating delays. This leads to high stress levels. To cultivate a more motivated and productive workforce, companies can focus on transforming management styles, implementing in-house soft skills training, promoting work–life balance initiatives and providing support for self-care and mental health.

This study tackles a critical gap in understanding data analytics skills in South Africa, while also providing novel insight into data professionals’ work experiences, job stressors and how they cope. The findings are not just informative; they are a call to action for universities to enhance their programmes to equip students to navigate the challenges of the modern workplace. Moreover, we highlight an urgent need for employers to prioritise the well-being of their data professionals. Investing in the mental health and soft skills of these individuals is not just beneficial — it is essential.

Acknowledgements

The authors thank all participants in this research project for their valuable contributions. This article is partially based on the author, W.C.’s article-based thesis entitled, ‘Using critical systems thinking to develop instructional guidelines for improving the employability of data analytics students’, toward the degree of Doctor of Philosophy in Science with Operational Research in the School for Mathematical and Statistical Sciences, North-West University, South Africa, with supervisor Prof. R. Goede, received 2025. This article forms part of one of the chapters in the article-based thesis.

The authors declare that they have no financial or personal relationships that may have inappropriately influenced them in writing this article.

W.C. was responsible for the study’s conception, design, data collection, data analysis and data visualisation. W.C. wrote and edited all the drafts of the manuscript. R.G. provided supervision and guidance for improving the manuscript.

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

The data that support the findings of this study are available from the corresponding author, W.C. upon reasonable request.

The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of any affiliated agency of the authors.

References