The Evolution Of The Data Engineer

William Wrigley our consultant managing the role
Posting date: 5/29/2019 2:12 PM
Every Data Science department worth its salt has at least one engineer on the team. Considered the “master builders,” Data Engineers design, implement and manage Data infrastructure. They lay down digital foundations and monitor performance.

At least, that’s what they used to do. 

Over the last few years, the role has shifted. Data Engineers have gone from mainly designing and building infrastructure, to a much more supportive and collaborative function. 

Today, a key part of the engineer role is to help their Data Analyst and Data Scientist colleagues process and analyse data. In doing so, they are contributing to improved team productivity and, ultimately, the company’s bottom line.

THE IMPACT OF THE CLOUD


In the past, a Data Engineer would often move data to and from databases. They’d load it in a Data Warehouse, and create Data structures. Engineers would also be on hand to optimise Data while businesses upgraded or installed new servers. 

And then along came the Cloud. 

The rapid dominance of cloud computing meant that optimisation was no longer needed. And as the cloud made it easy for companies to scale up and down, there was less need for someone to manage the data infrastructure.  

The collective adoption of the cloud has had a big impact on the function of Data Engineers. Because, provided a company has the funds, there is no longer the same demand for physical storage.

Freed from endless scaling requests, engineers have more time to program and develop. They also spend more time curating data for better analytics. 

AUTOMATING THE BORING BITS 


Less than a decade ago, if start-ups wanted to run a sophisticated analytics program, they’d automatically hire a couple of Data Engineers. Without them, Data Analysts and Data Scientists wouldn’t have any Data. The engineers would extract it from operational systems, before giving analysts and business users access. They might also do some work to make the Data simpler to interpret. 

In 2019, none of this extraction and transformation work is necessary. Companies can now buy off-the-shelf technology that does exactly what a Data Engineer used to do. As Tristan Handy, Founder and President of Fishtown Analytics, puts it: “Software is increasingly automating the boring parts of Data Engineering.” 

STILL SOUGHT-AFTER 


With automation hot on the Data Engineer’s tail, it can be tempting to ask whether they are still needed at all. 

The answer is: yes, absolutely.

When recruiting engineers, Data Strategist Michael Kaminsky says he looks for people “who are excited to partner with analysts and Data Scientists.” He wants a Data Engineer who knows when to pipe up with, “What you’re doing seems really inefficient, and I want to build something better.”

Despite the rise in off-the-shelf solutions, engineers still play a key role in the Data Science team. The difference is simply that their priorities and tasks have shifted. 

Today, innovation is the watchword. The best engineers are hugely collaborative, helping their teams go further, faster.

It’s an exciting time to be a Data Engineer. If you’re interested in this field, we may have a job for you. Take a look at our latest opportunities or get in touch with our expert consultants.  

Related blog & news

With over 10 years experience working solely in the Data & Analytics sector our consultants are able to offer detailed insights into the industry.

Visit our Blogs & News portal or check out the related posts below.

Hunted’s Ben & Danni On Cybersecurity

Hunted’s Ben & Danni On Cybersecurity

Ben Owen and Danni Brooke are the Co-Directors for the EMEA Practice at Fortalice Solutions, a leading global cyber security and intelligence operations company.  They travel globally to assist clients with their cyber security requirements, bespoke training needs, intelligence and investigations both online and physical and counter fraud training/consultation. They deliver and manage a portfolio of pro-active intelligence solutions to keep people, nations and businesses safe from threats and head up the EMEA operations.  Ben and Danni also feature on the hit Channel 4 show, Hunted and Celebrity Hunted which has been airing for over four years with another series set to be filmed this summer. I caught up with them recently to discuss the latest Fraud, tools and challenges for the Cybersecurity industry. Cybersecurity is an ever-changing landscape. What trends do you anticipate for the next 12 months and beyond? It is always difficult to pin down what the next real trend is going to be in the Cybersecurity space as adversaries are becoming ever more sophisticated.  What was once a very difficult process for skilled individuals is becoming more readily available to novices with advances in software, particularly those shared on the Dark Web. What is an inevitable threat trend in the next 12-months and beyond is the exponential rise in the Internet of Things (IoT).  With a world where everything is hooked up to the web, it is apparent that tech companies selling these devices are under immense pressure to get products to market. The need for speed could mean that some security principles and best practices may be overlooked.   As the UK encountered during the Mirai Botnet attack of 2016, a network of electronic devices acting in concert can cripple the internet or, worst case, become a weapon that could cause actual physical damage as well as cyber damage, power stations, hospital networks to name but a few.   How have Data & Analytics impacted the detection, and prevention, of cyber-crime? A company will have to protect themselves against an enormous amount of cyber threats every second.  A cyber-criminal will only need one successful attempt. Data & Analytics are proving successful in the fight against cyber-crime and their proactive and holistic approach is at keeping people and businesses safe.  Of course, it is Data that is being stolen, but very often Data can come to the rescue.  It helps in a number of ways, e.g. identifying anomalies in employee and contractor computer usage and patterns, detecting irregularities in networks, identifies irregularities in device behaviour (a huge advantage with the rise of the IoT). What one must remember, however, is the people behind the Data.  You can’t simply collect Data and assume you will be able to detect and respond with the right actions.  You need the people with the right analytical skills to sift through the Data, find the right signals and then react to the threat with an appropriate and timely response.   What tools and technologies do you think will become increasingly important in the fraud and cyber-crime landscape? Here at Fortalice we are investing a lot of time into coverage of the Dark Web.  We live in a rapidly changing digital landscape. Criminals, fraudsters, and others are now operating with more sophistication and anonymity. Where do they go to exchange fraudulent details and ideas about current victims? What medium do they use to discuss organisational targets or new ways of defrauding companies? The answer is the Dark Web.  Traditionally, companies fight fraud from the inside out. We want to change this landscape by accessing the entirety of the Dark Web, its pages, shady storefronts, and treasure troves of Data, and drawing on monitoring toolsets to give our clients a 360-degree resource for identifying adversarial communications and movements. It’s all about Internet coverage.  Wherever it is difficult to find – that’s where your threat will be.   A final point to this question is one of sharing tools and techniques.  A collaborative approach is always a good way of making sure the wider audience benefits.  We always work with our clients and offer other services and support outside of our remit to make sure they’re fully protected from a cyber and physical space.   What are the biggest security threats for businesses? Security is fundamentally broken because the design of many security solutions does not design for the human psyche.  Security solutions are bolted on, clunky, and hard to use but because security teams prioritise defending against easier cyber threats, they often don’t focus on the hardware side. The biggest risk to companies and individuals is always defined by the Data that is most important to you or to the business.  For individuals, this might be privacy or identity. For businesses, this could be customer Data, intellectual property, and the company’s money in the bank. The reality is that business executives can’t outspend the (cybersecurity) issue and they must be prepared. Cybersecurity no longer exists in a vacuum and it must be elevated to the conversations held in the boardroom and with senior leadership as well as entire divisions, departments, and organisations. For someone trying to get into security analytics, what skills do you think are key to being successful in the industry? The detail is in the name of the role.  A huge ability to interpret large amounts of technical Data is key to the role, as well as being able to assimilate what it means and how to action it.  Risk management is also key to this role.  Very often you will identify potential risks and you will have to triage those priorities on your own as co-workers won’t have the technical expertise to assist.  You will need to be able to communicate successfully to all levels of a workforce and last but by no means least – a good sense of humour!  When you think you have gotten to understand a new threat or vulnerability a new one will replace it within seconds.  Time to put the kettle on, smile, and get back to work with your analytical prowess.   Within fraud, it's well known that criminals are sharing their approaches, is this mirrored in cyber-security and if so, how is the industry combating this? Criminal collaboration is huge on the web.  First of all, there is no talent shortage for fraud rings or cybercriminals. There are no requirements for fancy university degrees or certifications and the crime ring pays for performance.  They don’t care what you look like, how you dress, or if you clock in during normal work hours. They care about getting the job done - hacking into and stealing information from others. Together they are sadly stronger and more effective.  On Dark Web forums, you will see fraudsters sharing and selling their ‘IP’ knowing that others will also contribute. That way they are all winners.  In the private world ideas equal money. That is of course not a bad thing for business, but it is bad for collaboration. Businesses generally don’t like to share ideas with one another because it has taken them lots of time and expense to get to their product or solution. As cliché as this comment sounds - we have to change this landscape for the greater good.  There are lots of smart government initiatives for national defences in cyber security and fighting high-end cyber-crime but seldom does this have a positive impact locally with smaller businesses.  There is a huge amount of information out there for individuals and advice, but we need to bridge the gap still between criminal collaboration and that of the good guys. If you could change one thing in the industry, what would it be? The mind set of security professionals that humans are the weakest link. We’re not! Humans are at risk because technology is by design, open.  I’d also change the mind set of those not in the Cyber Security industry.  All too often the severity of what is being reported is not taken seriously, nor are budgets set aside for cyber security issues.  That said, it is improving but there is a long way to go.  Ben and Danni spoke to Senior Consultant, Rosalind Madge. Get in touch with Rosalind or take a look at our latest job opportunities here.

Where Tech Meets Tradition

Where Tech Meets Tradition

If you’re lamenting the decline of handmade traditional products, cast your cares aside. There’s a new Sheriff in town and its name is, Tech. Just a generation ago, children would leave the farm or the family business, go to school, and then move on to make their place in the world doing their own thing. Away from family.  Today, the landscape has changed and those who have left are coming home. But this time, they’re bringing technology with them to help make things more efficient and more productive. Is Tech-Assisted Still Handmade? In a word, yes. Artists still make things “from scratch”, except now technologies allow them to not only see their vision in real-time, but their customers, too. Have you ever wondered what the image in your head might look like on paper or in metal? What about the design of prosthetic arms and healthcare devices by 3D printers? You’re still designing, creating.  But just like any new technology, there’s still a learning curve. Even for cutting-edge craftspeople who find that sometimes, the line between craftsmanship and high-tech creativity may be a bit of a blur. Not to mention the expense for either the equipment required or being able to offer art using traditional tools at technology-assisted prices. Somewhere between the two, there is a trade-off. It’s up to the individual to determine where and what that trade-off is. Life in the Creative Economy One of Banksy’s paintings shredded itself upon purchase at an auction recently. AI is making music and writing books. Augmented Reality, Virtual Reality, and Blockchain all have their place in the creative economy from immersive entertainment to efficient manufacturing processes. Each of these touches the way we live now. In a joint study between McKinsey and the World Economic Forum, 'Creative Disruption: The impact of emerging technologies on the creative economy', the organisations broke down the various technologies used in the creative economy and how they’re driving change. For example: AI is being used to distill user preferences when it comes to curating movies and music. The Associated Press has used AI to free up reporters’ time and the Washington Post has created a tool to help it generate up to 70 articles a month, many stories of which they wouldn’t have otherwise dedicated staff.Machine Learning has begun to create original content. Virtual Reality and Augmented Reality have come together as a new medium to help move people to get up, get active, and go play whether it’s a stroll through a virtual art gallery or watching your children play at the playground.  Where else might immersive media play out? Content today could help tell humanitarian stories or offer work-place diversity training. But back to the artisan handicrafts.  Artistry with technology Whilst publishing firms may be looking to use AI to redefine the creative economy, they are not alone. Other artists utilising these technologies include:  SculptorsDigital artistsPaintersJewellery makersBourbon distillers America’s oldest distiller has gotten on the technology bandwagon and while there is no rushing good Bourbon, but you can manage the process more efficiently. They’ve even taken things a step further and have created an app for aficionados to follow along in the process. Talk about crafted and curated for individual tastes and transparency. It may seem almost self-explanatory to note how other artisans are using technology. But what about distilleries? What are they doing? They’re creating efficiency by: Adding IoT sensors for Data Analytics collection Adding RFID tags to their barrels Creating experimental ageing warehouses (AR, anyone?) to refine their craft. Don’t worry, though. These changes won’t affect the spirit itself. After all, according to Mr. Wheatley, Master Distiller, “There’s no way to cheat mother nature or father time.” Ultimately, the idea is to not only understand the history behind the process, but to make it more efficient and repeatable. A way to preserve the processes of the past while using the advances of the present with an eye to the future. If you’re interested in using Data & Analytics to drive creativity, we may have a role for you. Take a look at our latest opportunities or get in touch with one of our expect consultants to find out more. 

RELATED Jobs

Salary

£70000 - £100000 per annum + bonus, pension, benefits

Location

City of London, London

Description

An opportunity to develop the data governance strategy for data's fastest risers within the consultancy space.

Salary

€70000 - €100000 per annum + BENEFITS

Location

Munich, Bayern

Description

Unser Kunde, ein modernes Beratungshaus im Bereich Big Data und Data Warehousing sucht derzeit nach einem erfahrenen Data Governance Consultant (m/f/d)!

Salary

£30000 - £45000 per annum + bonus, pension, benefits

Location

Birmingham, West Midlands

Description

An opportunity as a Data Governance Analyst for a leading, global financial services brand with a fantastic appetite for data.

Salary

800000kr - 1000000kr per annum

Location

Oslo

Description

Dette er en fantisktisk mulighet som leder av Data Governance for en av Norges ledende banker, hvor du vil lede sentrale data governance prosjekter!

recently viewed jobs