Accessibility Links

Data Engineers, the unsung heroes of Data Science

Data Engineer

Before you can build a house, you need a blueprint of its design and schematics. When you begin construction, you must first lay the foundation upon which it will be built. Tangible products taken step-by-step to create first a house and then a home. However, in the world of data science, companies seem to have skipped the blueprints and foundational aspects and gone straight for the aesthetics. But, how do you decorate a house before it’s built?

A house without a foundation becomes a house of cards and the same is true of data analysis. Before the data scientists can process and analyse data, first must come the engineers. The Data Engineers who lay the digital foundation and set the parameters, who create the data lakes and platforms, so the data analysts have something to make sense of. As high as the demand is for data scientists, the demand and the need, is even greater for data engineers, yet a shortage remains.

Where are the Data Engineers?


Data engineering jobs outnumber data scientist jobs nearly four to one according to a quick search on job boards such as Glassdoor and Indeed. Yet, the complex technical nature of data engineering to support data scientists takes more than a degreed education. Unlike data analysts, data scientists, and other data professionals who can land a mid-level job directly out of university, data engineers cannot.

Ultimately, it takes between five to ten years for mid-level data engineers to gain enough experience for practical application. As such, systems do not yet exist in schools and universities to supplement data engineers undergraduate or postgraduate degrees in preparation for real life work experience in the field. However, once the experience is gained, it can take a company who has hired a data engineer up to two years to catch up with its competition.

With the pace of change in the tech world, this can be detrimental to both the business and the data science teams. Therein lies the Catch-22, data engineers must have experience before they can be hired, but there is no way to learn outside of hands-on, real life application.

Why You Need to Add a Data Engineer to Your Data Science Team


A data science team is not complete without a data engineer. Why? Because just like building a house, grand schemes and ideas to solve complex business problems, must first have a foundation. Data engineers are that foundational support of experts who design, build, and maintain data-based systems and organizational operations.

Not only do data engineers lay the foundation upon which data can be built, analysed, and ultimately translated to business professionals, it must also be timely.  Timely data leads to more data and better predictions.

Data engineers are not completely siloed from data science teams, they are also responsible for deploying the code and models that are written by data scientists. For more on the reasons data engineering is more important than data science for companies today, check out this article from Captech Consulting.

Data Science Team Seeks Data Engineer


Companies know data drives business and they know the importance of data professionals. However, they may mistakenly assume either that their data teams can pick up engineering experience as they work their way through a project or they simply assume the titles are interchangeable.

In the world of data engineering, there is no entry level job. Experience trumps education in this field.

Like the once siloed data science team now integrated across the business with sales, marketing, and advertising departments, so must the role of data engineer be integrated. This is not a marriage of convenience, but of necessity in order to stay ahead of the competition. Together, your fully integrated data teams – data engineering and data science now on equal footing - will be able to help your business reach better predictions faster, making you a voice of authority in your discipline.

Your Turn: Route to the Role of Data Engineer


The route to the role of Data Engineer may seem daunting with the catch-22 that experience supersedes education. So, in the spirit of collaboration, we thought we’d ask for your thoughts and opinions on a few items of interest such as how we can educate aspiring data engineers and get them into companies faster. What kind of cross-training programs might businesses and schools employ to fill the shortage? What other backgrounds are we overlooking as businesses seek to find and engage this most critical role within their data science teams?

According to the website Datanami, 2018 will be the year of the data engineer. If this is you, then we may have a role for you.

Check out our current vacancies or contact Joshua Carter, Recruitment Consultant with a Data Engineering focus at +44 20 8408 6070 or email joshuacarter@harnham.com to learn more.