Data engineers, the unsung heroes of data science

Joshua Carter our consultant managing the role
Posting date: 5/21/2018 6:33 PM
Before you can build a house, you need a blueprint of its design and schematics. When you begin construction, you must first lay the foundation upon which it will be built. Tangible products taken step-by-step to create first a house and then a home. However, in the world of data science, companies seem to have skipped the blueprints and foundational aspects and gone straight for the aesthetics. But, how do you decorate a house before it’s built?

A house without a foundation becomes a house of cards and the same is true of data analysis. Before the data scientists can process and analyse data, first must come the engineers. The Data Engineers who lay the digital foundation and set the parameters, who create the data lakes and platforms, so the data analysts have something to make sense of. As high as the demand is for data scientists, the demand and the need, is even greater for data engineers, yet a shortage remains.


Where are the Data Engineers?

Data engineering jobs outnumber data scientist jobs nearly four to one according to a quick search on job boards such as Glassdoor and Indeed. Yet, the complex technical nature of data engineering to support data scientists takes more than a degreed education. Unlike data analysts, data scientists, and other data professionals who can land a mid-level job directly out of university, data engineers cannot.

Ultimately, it takes between five to ten years for mid-level data engineers to gain enough experience for practical application. As such, systems do not yet exist in schools and universities to supplement data engineers undergraduate or postgraduate degrees in preparation for real life work experience in the field. However, once the experience is gained, it can take a company who has hired a data engineer up to two years to catch up with its competition.

With the pace of change in the tech world, this can be detrimental to both the business and the data science teams. Therein lies the Catch-22, data engineers must have experience before they can be hired, but there is no way to learn outside of hands-on, real life application.


Why You Need to Add a Data Engineer to Your Data Science Team

A data science team is not complete without a data engineer. Why? Because just like building a house, grand schemes and ideas to solve complex business problems, must first have a foundation. Data engineers are that foundational support of experts who design, build, and maintain data-based systems and organizational operations.

Not only do data engineers lay the foundation upon which data can be built, analysed, and ultimately translated to business professionals, it must also be timely.  Timely data leads to more data and better predictions.

Data engineers are not completely siloed from data science teams, they are also responsible for deploying the code and models that are written by data scientists. For more on the reasons data engineering is more important than data science for companies today, check out this article from Captech Consulting.

Data Science Team Seeks Data Engineer

Companies know data drives business and they know the importance of data professionals. However, they may mistakenly assume either that their data teams can pick up engineering experience as they work their way through a project or they simply assume the titles are interchangeable.

In the world of data engineering, there is no entry level job. Experience trumps education in this field.

Like the once siloed data science team now integrated across the business with sales, marketing, and advertising departments, so must the role of data engineer be integrated. This is not a marriage of convenience, but of necessity in order to stay ahead of the competition. Together, your fully integrated data teams – data engineering and data science now on equal footing - will be able to help your business reach better predictions faster, making you a voice of authority in your discipline.

Your Turn: Route to the Role of Data Engineer

The route to the role of Data Engineer may seem daunting with the catch-22 that experience supersedes education. So, in the spirit of collaboration, we thought we’d ask for your thoughts and opinions on a few items of interest such as how we can educate aspiring data engineers and get them into companies faster. What kind of cross-training programs might businesses and schools employ to fill the shortage? What other backgrounds are we overlooking as businesses seek to find and engage this most critical role within their data science teams?

According to the website Datanami, 2018 will be the year of the data engineer. If this is you, then we may have a role for you.

Related blog & news

With over 10 years experience working solely in the Data & Analytics sector our consultants are able to offer detailed insights into the industry.

Visit our Blogs & News portal or check out the related posts below.

How to Succeed in Self-Service BI

How to Succeed in Self-Service BI

Business Intelligence, along with Business Analytics and Big Data, is one of the terms often associated with decision-making processes in organisations.  However, there is little discussion around the importance of what skills decision makers in your organisation need to use the technology efficiently.  In recent years, the development of user-friendly tools for BI processes, Self-Service BI are increasing. Self-Service BI is an approach to BI where anyone in an organisation can collect and organise data for analysis without the assistance of data specialists. As a result of this, many businesses have invested in comprehensive storage and information processing tools. However, many are beginning to find that they are not able to realise the gains of these investments as they were expecting, may often due to underestimating the difficulties of introducing these systems into the current processes and transforming existing knowledge into actual actions and decisions.  In a worst-case scenario, if left unplanned, Self Service BI can sabotage your successful BI deployment by cutting mass user adoption, impairing query performance, failing to reduce report backlogs, and increasing confusion over the “single truth”. To prevent this from happening, here are our top three tips for ensuring the right implementation of SSBI in your company: UNDERSTAND YOUR USERS’ NEEDS There are three major user areas for analytics tools: strategic, tactical and operational. The strategic users make few, but important decisions. The tactical users make many decisions during a week and need updated information daily. Operational users are often closest to the customer, and this group needs data in its own applications in order to carry out a large number of requests and transactions.  Understanding the different needs of each group is necessary to know what information should be available at each given frequency to help scale the BI solution.  HARNESS THE POWER OF ADVANCED USERS To ensure a successful BI deployment, utilising advanced users is key. Self-service BI is not a one-size fits all approach. Casual users usually don’t have the time to learn the tool and will often reach out to ‘Power Users’ to create what they need. Hence, these users can become the go-to resource for creating ad-hoc views of data. Power Users are the ideal advocates for your business’ self-service BI implementation and should be able to help spur user adoption.  UPGRADE INTERNAL COMPETENCIES  Our final tip for a successful implementation is to communicate the new tool thoroughly to the users.  It is highly unlikely that employees who have not been involved in the actual development project will immediately understand what the tool should be used for, who needs it, and what it should replace. By upgrading internal competencies, you can avoid becoming dependent on external assistance. Establishing a cross-organizational BI competence centre of 5-10 members, who meet regularly to share their experiences will help drives and prioritise future use of the tool. The added benefit of a successful implementation is that it will generate new ideas from users for how the organisation can use data to make better decisions. If you have the skillset to implement Business Intelligence solutions, we may have a role for you.  Take a look at our latest opportunities or get in contact with our team. 

Real Time Pricing - Coming to a store near you

Real Time Pricing - Coming to a store near you

Real-time pricing: coming to a store near you.Personal shopping is on the brink of taking on a whole new meaning. The advancement of mobile technology and the information held on individuals' shopping histories means product prices could soon adapt as shoppers walk up and down their supermarket aisle.Gone are the days of retailers only being able to actively manage the price of a small number of products once a week. Algorithmic pricing and real-time competitive pricing data allows the changing of product prices on the fly.Amazon is at the forefront of such "real-time pricing" initiatives, which have traditionally been the preserve of online-only retailers.However, brick-and-mortar retailers in the US are showing their UK counterparts the limitless possibilities when it comes to dynamic pricing.Independent consumer electronics retailer Abt Electronics pipes competitive pricing data gathered by Dynamite Data into its point-of-sale systems to allow staff to negotiate prices at the point-of-sale, according to Dynamite Data chief executive Diana Schulz.Meanwhile, another one of Dynamite Data’s unnamed clients uses electronic shelf labels and re-prices every product in their stores each morning based on the prices of its rivals.The ability to change prices dynamically is not simply the preserve of all-powerful brands such as Walmart or Target either.Schulz explained that her company has "seen these types of technologies in both large and mid-sized retailers" despite the "investment in technology and competitive data that is typically needed".Commercial sensitivitiesBack in the UK things are not quite as close to a Minority Report-style personalized shopping experience.Even online-only specialists Shop Direct and Ocado claim they do not engage in real-time pricing, while those that do heavily use real-time data to adapt their prices such as the airline brands are reluctant to discuss the issues.EasyJet declined to comment when contacted because of commercial sensitivities around discussing pricing-related issues.Grocers Tesco, Asda and  Sainsbury’s have all claimed they do not engage in real-time pricing, with the latter two both citing the logistical difficulties in aligning such a strategy across their physical stores and online presence.A Sainsbury’s spokesman claims real-time pricing would result in "chaos", while an Asda spokeswoman saying such a strategy would be a "nightmare".Yet, despite such a negative perspective from UK brands, experts are confident real-time pricing will arrive on these shores sooner or later.Simon Spyer, a partner of VCCP data arm Conduit who began his career working on the Sainsbury's Nectar business, believes the UK will begin to see "more and more" of matching rivals’ prices dynamically, particularly in the grocery and electrical sectors.He explained that real-time pricing is likely to affect "anything where the product is largely commoditized" and in instances where the only way retailers can differentiate that product is by "being really keen on price".Electronic labelsAs it stands the major barrier for implementing "real-time pricing" in-store is changing the prices to match the online price, a hurdle that could be removed by the electronic shelf labels being pioneered in the US.Schemes like Tesco Price Promise and Asda Price Guarantee already use real-time data to 'price match'In the UK various retailers have dipped their toes into the water when it comes to electronic shelf-labeling including a Nisa Local store in Shrewsbury that launched a trial in August last year to carry out automatic pricing and timed promotional updates, alongside QR codes and meal deals.Tesco has also experimented with electronic labeling on various occasions with trials in 2006 and 2008, but the retail giant has yet to combine real-time pricing with its electronic labels.Spyer claims "the capability is definitely there both online and offline – it is whether there is a business rationale for investing in it".However, with major UK supermarkets lacking a pressing reason to implement real-time pricing, that investment may be slow in arriving, argues Kaye Coleman, the founder of price consultancy Ripe Strategic.Coleman explains: "The supermarkets already do price matching – it is not so sophisticated but price matching is already happening".Schemes including the Tesco Price Promise, the Asda Price Guarantee and the Sainsbury’s Brand Match currently use real-time data to "price match" by offering money off the next shop.A cynic could argue the supermarkets should knock money off at the till rather than relying on customers to redeem their vouchers at the next shop, but such an action could hit the companies' bottom line.Mobile sophisticationThe growing sophistication of mobile marketing is also likely to revolutionize the way brands approach their price matching."If you can come up with a value proposition where I check-in [on my mobile] when I walk through the store for the first time and that presents me with a personalized experience based on my purchase history then I could see the benefit for a customer and a retailer," said Spyer.The trick for retailers is persuading customers to adopt such behavior, but the offer of being delivered ever-changing personalized price offers and messages in-store is a compelling proposition.Personalization is already a priority for retailers. Sainsbury’s uses anonymized shopping data gathered from the Nectar card to personalize offers.The levels of personalization offered by Sainsbury’s are increasingly complex. If a female customer buys folic acid they will be sent promotions on other pregnancy-related supplements during the pregnancy period and offers on nappies further down the line.UK retailers are sure to keep a close eye on developments over the Atlantic, with Schulz claiming she knows of clients that are piloting technologies that enable in-store personalized discounts.The challenges on the high-street mean there will inevitably be more casualties, but real-time pricing does not have to be the sole preserve of online-only retailers.Innovative ways of manipulating real-time data could be the shot in the arm the high-street retail industry so desperately needs.This article was first published on marketingmagazine.co.ukClick here for the article on the web.

RELATED Jobs

Salary

£50000 - £75000 per annum + + Benefits

Location

City of London, London

Description

This is the leading company in the online food retail space operating in the UK They are seeking an Infosec Team lead

Salary

£50000 - £60000 per annum

Location

City of London, London

Description

*SENIOR INSIGHT ANALYST - CENTRAL LONDON - SQL & PYTHON/R - UP TO £60,000*

Salary

£42 - £65000 per annum + bonus and benefits

Location

London

Description

A well known marketplace app are seeking a Senior Analyst.

Salary

£36000 - £37000 per annum + Yes

Location

Milton Keynes, Buckinghamshire

Description

Junior Data Scientist, London United Kingdom.

recently viewed jobs