Every Data Science department worth its salt has at least one engineer on the team. Considered the “master builders,” Data Engineers design, implement and manage Data infrastructure. They lay down digital foundations and monitor performance.At least, that’s what they used to do. Over the last few years, the role has shifted. Data Engineers have gone from mainly designing and building infrastructure, to a much more supportive and collaborative function. Today, a key part of the engineer role is to help their Data Analyst and Data Scientist colleagues process and analyse data. In doing so, they are contributing to improved team productivity and, ultimately, the company’s bottom line.
THE IMPACT OF THE CLOUDIn the past, a Data Engineer would often move data to and from databases. They’d load it in a Data Warehouse, and create Data structures. Engineers would also be on hand to optimise Data while businesses upgraded or installed new servers. And then along came the Cloud. The rapid dominance of cloud computing meant that optimisation was no longer needed. And as the cloud made it easy for companies to scale up and down, there was less need for someone to manage the data infrastructure. The collective adoption of the cloud has had a big impact on the function of Data Engineers. Because, provided a company has the funds, there is no longer the same demand for physical storage.Freed from endless scaling requests, engineers have more time to program and develop. They also spend more time curating data for better analytics.
AUTOMATING THE BORING BITS Less than a decade ago, if start-ups wanted to run a sophisticated analytics program, they’d automatically hire a couple of Data Engineers. Without them, Data Analysts and Data Scientists wouldn’t have any Data. The engineers would extract it from operational systems, before giving analysts and business users access. They might also do some work to make the Data simpler to interpret. In 2019, none of this extraction and transformation work is necessary. Companies can now buy off-the-shelf technology that does exactly what a Data Engineer used to do. As Tristan Handy, Founder and President of Fishtown Analytics, puts it: “Software is increasingly automating the boring parts of Data Engineering.”
STILL SOUGHT-AFTER With automation hot on the Data Engineer’s tail, it can be tempting to ask whether they are still needed at all. The answer is: yes, absolutely.When recruiting engineers, Data Strategist Michael Kaminsky says he looks for people “who are excited to partner with analysts and Data Scientists.” He wants a Data Engineer who knows when to pipe up with, “What you’re doing seems really inefficient, and I want to build something better.”Despite the rise in off-the-shelf solutions, engineers still play a key role in the Data Science team. The difference is simply that their priorities and tasks have shifted. Today, innovation is the watchword. The best engineers are hugely collaborative, helping their teams go further, faster.It’s an exciting time to be a Data Engineer. If you’re interested in this field, we may have a job for you. Take a look at our latest opportunities or get in touch with our expert consultants.