Data engineers, the unsung heroes of data science

Joshua Carter our consultant managing the role
Posting date: 5/21/2018 6:33 PM
Before you can build a house, you need a blueprint of its design and schematics. When you begin construction, you must first lay the foundation upon which it will be built. Tangible products taken step-by-step to create first a house and then a home. However, in the world of data science, companies seem to have skipped the blueprints and foundational aspects and gone straight for the aesthetics. But, how do you decorate a house before it’s built?

A house without a foundation becomes a house of cards and the same is true of data analysis. Before the data scientists can process and analyse data, first must come the engineers. The Data Engineers who lay the digital foundation and set the parameters, who create the data lakes and platforms, so the data analysts have something to make sense of. As high as the demand is for data scientists, the demand and the need, is even greater for data engineers, yet a shortage remains.


Where are the Data Engineers?

Data engineering jobs outnumber data scientist jobs nearly four to one according to a quick search on job boards such as Glassdoor and Indeed. Yet, the complex technical nature of data engineering to support data scientists takes more than a degreed education. Unlike data analysts, data scientists, and other data professionals who can land a mid-level job directly out of university, data engineers cannot.

Ultimately, it takes between five to ten years for mid-level data engineers to gain enough experience for practical application. As such, systems do not yet exist in schools and universities to supplement data engineers undergraduate or postgraduate degrees in preparation for real life work experience in the field. However, once the experience is gained, it can take a company who has hired a data engineer up to two years to catch up with its competition.

With the pace of change in the tech world, this can be detrimental to both the business and the data science teams. Therein lies the Catch-22, data engineers must have experience before they can be hired, but there is no way to learn outside of hands-on, real life application.


Why You Need to Add a Data Engineer to Your Data Science Team

A data science team is not complete without a data engineer. Why? Because just like building a house, grand schemes and ideas to solve complex business problems, must first have a foundation. Data engineers are that foundational support of experts who design, build, and maintain data-based systems and organizational operations.

Not only do data engineers lay the foundation upon which data can be built, analysed, and ultimately translated to business professionals, it must also be timely.  Timely data leads to more data and better predictions.

Data engineers are not completely siloed from data science teams, they are also responsible for deploying the code and models that are written by data scientists. For more on the reasons data engineering is more important than data science for companies today, check out this article from Captech Consulting.

Data Science Team Seeks Data Engineer

Companies know data drives business and they know the importance of data professionals. However, they may mistakenly assume either that their data teams can pick up engineering experience as they work their way through a project or they simply assume the titles are interchangeable.

In the world of data engineering, there is no entry level job. Experience trumps education in this field.

Like the once siloed data science team now integrated across the business with sales, marketing, and advertising departments, so must the role of data engineer be integrated. This is not a marriage of convenience, but of necessity in order to stay ahead of the competition. Together, your fully integrated data teams – data engineering and data science now on equal footing - will be able to help your business reach better predictions faster, making you a voice of authority in your discipline.

Your Turn: Route to the Role of Data Engineer

The route to the role of Data Engineer may seem daunting with the catch-22 that experience supersedes education. So, in the spirit of collaboration, we thought we’d ask for your thoughts and opinions on a few items of interest such as how we can educate aspiring data engineers and get them into companies faster. What kind of cross-training programs might businesses and schools employ to fill the shortage? What other backgrounds are we overlooking as businesses seek to find and engage this most critical role within their data science teams?

According to the website Datanami, 2018 will be the year of the data engineer. If this is you, then we may have a role for you.

Related blog & news

With over 10 years experience working solely in the Data & Analytics sector our consultants are able to offer detailed insights into the industry.

Visit our Blogs & News portal or check out the related posts below.

Marketing Analytics - Then, Now & In the Future: A Q&A with Sarah Nooravi

We recently spoke to Sarah Nooravi, an Analytics professional with a specialism in Marketing who was named one of LinkedIn’s Top Voices in Analytics.  Sarah found herself working in Analytics after being attracted to the culture, creativity and the opportunity to be challenged. Having spent the first four years of her career working within the Marketing space, she has seen a real transition in the way that Analytics and Data Science has informed Marketing decisioning.  “I started my career in a Marketing agency within the entertainment industry, at the time it was doing things that most of the entertainment industry hadn’t considered doing yet”.  At the start of her career she’d meet entertainment giants with advertising budgets of millions of dollars who were, at the time, making mostly gut decisions with how to approach campaigns. “It was common that I’d hear, ‘I think our audience is females over the age of 35 with a particular interest and we should just target them’” she expands.  However, agencies quickly recognised the need for something more Data-driven. Entertainment businesses were going too narrow and were misunderstanding their audiences. The next step was to embed into these businesses the insights from a greater variety of sources, including social media, and to introduce more testing. That translated into a better media buying strategy that could be continuously optimised. It was a big step forward in the utilisation of Data within this realm and its clear focus on ROI.  Suddenly, the market was changing, “There was a massive spike of agencies popping up and claiming to leverage Data Science and Machine Learning to provide better optimisations for entertainment companies, mobile gaming – you name it. There was a huge momentum shift from using these gut decisions to leveraging agencies that could prove that”.  What she saw next seemed only natural, with more agencies offering Data-driven optimisation, companies looked to develop this capability internally. Sarah elaborates; “Now I am seeing these companies starting to take ownership of their own media buying and bringing the Marketing and Data Science in-house”. This shift in-house has been propelled by the major players, companies like Facebook, Google and Nooravi’s own company, Snapchat, working directly with companies to help them optimise their campaigns. This shift has changed the landscape of Marketing Analytics, specifically within the advertising space. Sarah explains, “You no longer need an agency to optimise your, for example, Facebook campaigns, because Facebook will do it for you. They are minimising the number of people behind the campaigns. You give up a little of your company’s Data for a well optimised campaign and you don’t have to hire a media buyer. There is definitely a movement now to becoming more Data-driven. Companies are really leveraging A/B tests and also testing out different creatives”.  It is this change in strategy that is seemingly taking the Marketing Analytics challenge to the next level. With opportunities to pinpoint specific audiences, companies are using their Data to understand how to approach their content, take the opportunity to experiment, and to find out what it takes to resonate with their audience. Sarah has seen the potential of this first hand: “We are starting to see a lot of AR and VR. There are meaningful ways to engage with technology to connect with the world. Moving forward, content will have to become more engaging. People’s attention spans are becoming shorter and with each decision someone makes it is changing the direction of content in the future. There has been a massive shift from static images to video advertisement and, more recently, from video into interactive video like playable adverts. People want to engage with adverts in order to understand a company’s message”.  It is within this space that she sees a gap for the future of ROI positive advertising:  “The biggest issue that I find with the creative and the content is that the value add is missing. The resonance with the brand or company, their values and mission is what is missing. Analytics alone cannot fix that. You need to understand what the company stands for, people want to connect with brands because of what they stand for – whatever it is. Especially in a time like we are dealing with right now, a pandemic, advertising spending has gone down. However, maybe there is a way to properly message to people that would resonate. Not that you want them to buy your stuff but maybe right now is the perfect time to do outreach and to help people understand your brand”. The ability to understand and predict customer behaviour is evolving, but with that, so is the customer. Whereas at the moment, you can build out experiments, you can create models that will be able to, as Sarah explains, “in real-time decide whether a user’s behaviour is indicative of one that is going to churn” and then try and create offers to increase retention.   This is the challenge of the current analytics professional – our behaviours in a global pandemic have shifted consumers into a new world. Now working for Snap Inc, she sees the potential of this from a new perspective. Naturally, like most social media channels and communication technologies, they have seen an increase in usage over the last month.  “People are wanting to communicate more as we are forced to social distance. However, we are seeing different regions engaging a lot more heavily. For example, it's Ramadan right now, people want to share those moments with one another and at the moment the way that they are having to do that is changing”.  So, it will be a question for all those required to predict behaviours to determine how many of these new lines of communication, these new habits, will have evolved. Once people are out of quarantine, are they going to continue to utilise the apps, games, social channels in the same way that they are currently? It certainly is going to be something that many within the marketing analytics space will be trying to forecast.  If you’re looking to take your next step in Marketing Analytics, or are looking to build out your team, Harnham may be able to help.  Take a look at our latest opportunities or get in touch with one of our expert consultants to find out more. 

How To Write The Perfect Tech Job Description

It’s a challenge finding the right Data & Analytics candidate. Add in the number of companies fighting for that perfect profile and for many it can seem like an uphill battle. But there’s a simple way to cut through the noise; better job descriptions.  As a specialist recruitment agency within the Data & Analytics space, we have seen a real variety of job descriptions over the years, from the bright and innovative to the long and technical. And it may surprise you to learn that candidates still ask regularly to see official job descriptions and it is part of their decision-making process.  Unfortunately, they are also often a part of the recruitment process that can be rushed or created from out-of-date previous descriptions. There are some real benefits, however, to putting the time and effort required into creating something fresh.   If you’ve recruited a role like a Data Scientist before, you know that the problem isn’t usually getting enough candidates through the door, it’s about getting the right ones. A well-crafted job description leads to better quality applicants. It also helps those candidates become more engaged and excited about your business.  So, with that in mind, here are our five top tips for businesses looking to help their role stand out from the crowd.  CHECK YOUR JOB TITLE  You might think that calling your BI Analyst a ‘Data Ninja’ is going to get you the top talent, but it would probably mostly cause confusion. It is important that you align the job title to a clear and market relevant job title. Often internal job titles can be the biggest blocker in aligning your vacancy to the market.  Consider changing the job title for external purposes to make it more closely aligned to the market. Here are some common examples:  An AVP Analyst within a Marketing Analytics team is more closely aligned to a Senior Marketing Analyst. A Data Scientist job title aligned to a role with no machine Learning or algorithmic development may be better titled a Statistical Analyst.  CREATE A COMPELLING JOB RUN-THROUGH  Our consultants agreed unanimously that one of the weakest areas of job descriptions tends to be the more detailed description of what the role actually is. Too often job descriptions just list lots of different responsibilities, but these are often very generic or basic.  Before starting to write the job brief, ask members of your team that do the role already – what gets them excited?  You will likely find that it has to do more with the types of projects i.e. the application of technical elements, that appeals most to candidates. If you can, bring the role to life in a meaningful way. For example, relating it to projects that your team has done is a really enticing method of exciting a candidate about the potential of the role. Create A Tailored Experience Section Uninspiring job descriptions often have long lists of key skills required, often with irrelevant skills included. Keep your requirements to around 5 or 6 key bullet points, asking yourself what the most important requirements are and clearly laying those out.  On top of that often companies get too focused on requesting years of experience. We strongly discourage companies from specifying years of experience in a job advert as, within the UK, most European countries and a number of US states this is classified as age discrimination. Instead of including years of experience, carve out what it is that you want your ideal candidate to have done before instead, this will often correlate to their experience level. For example: 5+ years' experience in a Marketing Analytics could easily be transformed to Proven commercial experience in a Marketing Analytics environment with exposure to pre and post campaign analysis, customer analysis,  customer segmentation and predictive modelling.  DON’T FORGET TO SELL YOURSELVES Another key area where many companies fall down is effectively selling their opportunity and company to the prospective candidates. Whether an active or passive job-seeker, candidates are likely deciding whether this is the right fit for them based on what they are reading. Many job descriptions completely forgo any type of sales pitch above an initial description of what the company does, perhaps because they expect the candidate to know them and want them.  These are the areas we’d suggest bringing to life to effectively sell your opportunity: Writing in your brands personality. Consider the right tone of voice to match your company culture and style of working. Introduce yourself. Whether you’re a brand name or not, use this chance to actually tell people about what you really do and what you really stand for. Share what it’s like to work for the company. Include the culture, work environment, targets, challenges and of course reference to perks and benefits on offer too. Consider the candidate. What appeals to a talented Data Scientist will differ from what appeals to an HR professional. Make sure you tailor your overall pitch to the type of candidate you are seeking.  WORK ON THE LOOK AND FEEL  A little effort on the aesthetic look of your job description an go a long way.  On top of a nice overall look, keep the length to a maximum of 1.5 pages. Utilise bullet points and bold formatting to keep the description some-what ‘skimmable’.  If you’re looking to hire a Data & Analytics professional, Harnham can help. Get in touch with one of our expert consultants to find out more. 

RELATED Jobs

Salary

US$200000 - US$225000 per year

Location

New York

Description

Are you passionate about helping uncover insights for the whole business by leveraging advanced analytics?

Salary

US$115000 - US$130000 per annum

Location

Chicago, Illinois

Description

This is an ideal opportunity for a Digital Analytics specialist currently working at a network agency who wants to take more leadership responsibility.

Salary

US$120000 - US$150000 per annum

Location

New York

Description

This is a great chance for a developer who has great hands-on coding skills to lead a large scale project.

recently viewed jobs