A recent study has revealed that 78% of data engineers wish their work had access to a therapist. On the other hand, the burnout-inducing workload has resulted in skyrocketing compensation as a result of fierce rivalry for data specialists.

Aside from the growing demand for therapists, the 2021 Data Engineering Survey, which DataKitchen and data.world collaborated, released October 19, contained a few other eye-opening numbers. For example, 97% of the 600 data engineers polled that they were burned out with another 70% saying they plan to quit within the next 12 months.

According to the poll, several reasons contribute to data engineers’ despondency, including the requirement to spend a lot of time:

• Playing whack-a-mole with faulty data pipelines;
• Finding and correcting data issues on a regular basis;
• Dealing with corporate users’ unrealistic requirements and expectations;
• Observing ever-stricter data regulations and governance standards;
• When they’re supposed to be playing, they’re working;
• When analytics programs go awry, as happens all too frequently, you will be “shamed and blamed.”

Bryon Jacob, CTO, and co-founder of data.world, said their survey acts as a wake-up call to data organizations because data engineers are calling for change. That entails establishing reliable, efficient, and repeatable workflow rules that will develop analytics cooperation and productivity while healing the sanity of data engineers.

While DataKitchen and data.world claim that their respective DataOps and data catalog products can assist boost data engineer morale, automation isn’t a miracle cure. Although automation can help, skilled individuals will continue to be necessary for big data success.

Sadly, obtaining experienced data experts is becoming increasingly difficult. According to research released earlier this year by the recruiting firm Burtch Works, 81% of US organizations planned to expand their data science, analytic, and engineering teams in the third and fourth quarters of the year.

“It’s no surprise to us that with the economic restart kicking into high gear and data science and analytics being a key element for digital transformation efforts (which are crucial to maintaining a competitive advantage in most industries), we’re seeing many companies keen to beef up their teams this year,” Burtch Works stated in a blog post.

Harnham, a UK-based data and analytics recruitment with offices in San Francisco, New York, and Arizona, can contribute more insight into the situation of the data professional. The organization monitors the supply and demand for data specialists, and the balance is currently out of whack.

Recent research indicates that organizations in the United States are having problems filling open data and analytics positions, according to Harnham’s Head of Marketing Owain Wood. In September 2020, for example, 33,000 people were looking for a specific data job. The number had reduced to roughly 15,000 by September 2021.

Wood tells Datanami that the number of people looking for work has roughly halved. “Vacancies are rising higher than the job seekers are looking for roles, which is really, really crazy to see.”

Harnham benefits from the rising competition for data specialists. When traditional methods, such as job postings, fail to attract talent, companies resort to headhunters like Harnham to fill unfilled positions. Harnham has increased the number of data experts it has placed in the last year, according to Owain.

They only see a couple of active applicants per position in some specialty roles where they were used to seeing 10 to 15. As a result, their headhunting services are in high demand, as there is a war for skilled talent.

Companies with unfilled positions in data analytics, engineering, and data science are seeing their staff expenditures rise as a result of the conflict. According to its most recent annual compensation survey, entry-level employment in advanced analytics currently pays between $90,000 and $100,000, with executive-level positions paying up to a quarter-million dollars per year.

Some jobs pay far more than others. For entry-level roles in computer vision and robotics, for example, experts can earn $125,000, while experienced vice presidents can earn up to $350,000.

“The amount of money that’s being thrown around is insane,” Wood says. “I’ve seen some jobs increased salaries maybe 100%, 150% over the past six months.”

These pay scales aren’t available in every business. For example, demand for industries affected by COVID, such as travel and hospitality, is still low. Demand, on the other hand, is rapidly increasing in other areas.

If you have touchpoints in your track record, in your background, you’re likely to get scooped up left, right, and center for high-demand industries like life sciences and healthcare, Wood says. But finding people in those places is quite difficult.

When COVID was front and center in 2020, data professionals in the United States were grateful to have a job. However, as a result of burnout and the temptation of better income at other jobs, people are quitting their positions in droves.

That is supported by data from the Bureau of Labor Statistics. According to the most recent BLS statistics, 4.3 million Americans quit their jobs in August, a new high. The number of job opportunities in the information sector, which includes data professionals, more than quadrupled from 78,000 in August 2020 to 154,000 in August 2021. However, according to the BLS statistics, the number of new employees decreased from 111,000 in August 2020 to 97,000 in August 2021.

Workers in the United States are in a state of flux across industries. More than half of US workers want to hunt for a new job in the next 12 months, according to a Bankrate study conducted in August. The workers most inclined to leave for greener pastures are younger and earn less money. However, 45 percent and 33 percent of Gen X and Baby Boom cohorts, respectively, said they will be seeking a new career.

Taken together, the evidence suggests that the data profession, if not all American professions in general, is at a fork in the road. Data engineers and data scientists are in high demand, but they are becoming increasingly dissatisfied with their jobs’ rising expectations. Over the next year, a huge number of data professionals appear to be on the verge of quitting their positions. If they want to stay in the profession, this might enhance their pay, even more, especially considering the overall paucity of skill.

To be sure, it’s a mixed bag. Individual data professionals have never had it better in terms of salary, but they’ve also never had it better in terms of demands. The demand for businesses to become “data-driven organizations” has never been greater, but the path to get there is not easy, given the scarcity and high cost of the talented personnel required to lead enterprises to the promised land of data.

Individuals appeared to be positioned to profit from the imbalance in the short term. Automation of the kind that data.world and DataKitchen have in data engineering–as well as a slew of other processes in data analytics and data science–seems destined to happen in the long run. Fasten your seatbelts, because the AI revolution is here, and it’ll be a rocky trip.

Source: Datanami