Route to the role of Data Engineer

Joshua Carter our consultant managing the role
Posting date: 5/21/2018 7:22 PM
Do you like breaking things down to see how they work? Do you want to build something that helps solves problems and can make lives better? Are you a puzzle solver curious about the world around you with a knack for mathematics? Do you prefer to work behind the scenes or front of stage? If you want to be the person behind the curtain, then this is your year. The year of the Data Engineer is here. 

In last week’s article, we talked about Data Engineer as the unsung hero of the data science world and briefly touched on route to the role of engineer. Though experience supersedes education, you do need the basics – a bachelor’s degree in computer science, data science, applied math, physics, statistics, software/computer engineering which can lead to a Master’s in Data Engineering and to cement your knowledge – fellowships and professional organisations are now available around the world. In today’s article, we’ll cover a few options. 

Lay Your Educational Foundation

Computer Science, Data Science, and Engineering programs abound in university today, but no school can really teach big data skills. It’s too focused. Most schools today offer general purpose tech education with a focus on web development or backend systems. And here begins that Catch-22. Though experience supersedes education, you still need a framework from which to build. 

More often than not, if you type Data Engineer into Google looking for education programs, you’ll get undergrad opportunities for Data Science. However, that’s not to say a Bachelor’s in Data Science can’t lead you to a Master’s in Data Engineering. So, how do you get from point A to point B? Here are a few suggestions: 

Beef up your skills with specific certifications for the languages businesses need – Scala, Python, and Java 

Take courses in data engineering technology: Hadoop, Spark, AWS, GCP, Azure etc.  

Join a professional organization for Data Engineers such as The Data Warehousing Institute  (TDWI) or the Institution of Engineering and Technology (IET) – here you’ll find articles, resources, and a network of mentors ready to offer advice and suggestions. 

Apply for a fellowship  with ASI Data Science – an 8-10 week intensive project with one of their partner companies to solve real-world business problems using Data Science or Data Engineering skills. If you’re a postgrad or higher, this a perfect opportunity to build your portfolio. 

Boost Your Data Engineering Resume with These Tips

In the world of data engineering, it’s important to highlight the details. 
  • Be specific: Companies will be more interested in interviewing you if you can clearly outline why/what you have used different technology for. Keep this punchy and concise, and outline your in-put with said technologies
  • Outline projects you’ve worked on 
  • Detail the technologies you’ve used 

David Bianco, a Data Engineer with Urthecast, offers the following advice to data engineering students. 
  • Be fluent in the languages and tools you use to get the job 
  • Understand the concepts behind what you’re doing 
  • Get involved with a community – meetup.com, hackathons, and other groups in your area are great places to get started. 

If you’re interested in switching your career  to Big Data, check out Jessen Anderson’s new e-book, The Ultimate Guide to Switching Careers to Big Data -- Upgrading Your Skills for the Big Data Revolution. 

Your Turn: Route to the Role of Data Engineer

Our data driven world moves at lightning speed and it can be hard to keep up. If you’re a Data Engineer, we want to hear your story. 

What was your route to the role?

What kind of cross-training programs might businesses and schools employ for future Data Engineers?
  
What other backgrounds are we overlooking as businesses seek to find and engage this most critical role within their data science teams?
  
What can we, as recruiters do to engage qualified candidates ready for their next role in the world of data and analytics? 

Related blog & news

With over 10 years experience working solely in the Data & Analytics sector our consultants are able to offer detailed insights into the industry.

Visit our Blogs & News portal or check out the related posts below.

2020: The Year of the Data Engineer

Data Engineers are the architects of Data. They lay the foundation businesses use to collect, gather, store, and make Data usable. Each iteration of the Data as it moves along the pipeline is cleaned and analysed to be used by Data professionals for their reports and Machine Learning models. A ROLE IN HIGH DEMAND Even as businesses reopen, reassess, and for some, remain remote, the demand for Data Engineers is high. Computer applications, Data modelling, prediction modelling, Machine Learning, and more need Data professionals to lay the groundwork to help businesses benefit in today’s Data-driven culture. The word gets thrown around a bit, but when the majority of business has moved online, Data-driven is the name of the game. Having a Data plan, a Data team, and all aligned with your business strategy is imperative to the way business is done today. This type of innovation can offer insight for better business decisions, enhance customer engagement, and improve customer retention without missing a beat.  Without Data Engineers, Data Scientists can’t do their jobs. Understanding the amount of Data, the speed at which is delivered, and its variety need Engineers to create reliable and efficient systems. Like many Data professional jobs, even still in 2020, Data Engineers are in high demand. Yet a skills shortage remains. This has created an emerging field of professionals from other backgrounds who are looking to take on the role of Data Engineer and fill the gap. Whether by necessity or design, these individuals build and manage pipelines, automate projects, and see their projects through to the end result. CAREER OPPORTUNITIES OUTSIDE THE NORM As this growing trend emerges, it has created career opportunities for those with experience outside the normal channels of Data Engineering study. While it might involve individuals from backgrounds such as software Engineering, Databases, or something similarly IT-related, some businesses are upskilling their employees with talent. Rapid growth, reskilling, upskilling, and ever-constant changes still leave businesses with a shortage of Data Engineers to meet the demand. It’s critical to fill the gap for success. According to LinkedIn’s 2020 Emerging Jobs Report, Data Engineering is listed in the top 10 of jobs experiencing growth. THREE STEPS TOWARDS BECOMING A DATA ENGINEER This is a vital role in today’s organisations. So, if you’re in the tech industry and want to take a deeper dive into Data as a Data Engineer, what steps can you take? This is a time like no other. There’s time to assess your goals, take online classes, and get hands on with projects. Though having a base of computer science, mathematics, or business-related degree is always a good start. Be well-versed in such popular programming languages such as SQL, Python, R, Hadoop, Spark, and Amazon Web Services (AWS).Prepare for an entry-level role once you have your bachelor’s degree.Consider additional education to stay ahead of the curve. This can include not only professional certifications, but higher education degrees as well. The more experience, hands-on as well as academic, you have the more in demand you’ll be as a Data Engineer. Data scientists might be the rockstars of Data, but Data Engineers set the stage. As business processes have shifted online, looking for your next job has become more daunting than ever before. If you’re looking for your next opportunity in Data, take a look at our current jobs or get in touch with one of our expert consultants to find out more. 

WE HAVE TO TEACH SPECIALISATION, WE CAN’T EXPECT IT: A Q&A WITH VIN VASHISHTA

We recently spoke to Vin Vashishta, a consulting Data Scientist and Strategist who was named one of LinkedIn’s Top Voices in Data Science.  Having started off in the tech world 25 years ago and progressing from web design and hardware installation to Business Intelligence Analytics, Vin found for many years that enterprises were reluctant to adopt AI technologies and embrace the value of Data. In fact, it wasn’t until the beginning of the decade just passed that companies started to think about their Data more strategically and the world of Data Science was born, albeit hesitantly:  “When I first started, it was a lot of experimentation, everyone wanted a proof of concept,” he says. “A lot of work was creating models that could go from whiteboard to production and productise and show their value.” However, it wasn’t until halfway through the decade that he began to see businesses who had adopted Machine Learning move away from experimentation into incorporating it more deeply into their companies, relying more on analytical and optimisation models to make strategic business decisions.  “After that, in about 2017/2018 the maturity changed. It went from being a one off implementation to it being a comprehensive tool within an organisation where we have full lifecycles of model implementation and full models that were full views of the system. The key component of development was allowing users to access a small part of the system to do their job better without having to understand the whole thing. And that’s where we are now. We have this applied Deep Learning and we are seeing, especially this year, attempts to optimise that, make things go faster and make them more repeatable.” But, as we all know, with great power comes great responsibility: “There’s this whole depth we are getting into, the expectations are so much higher, people don’t just expect it to work they expect it to work the way they want it to and in a way they can adopt.” So, with so much expected and required of Data Scientists in 2020, building the right team is more important than ever. However, many businesses, Vin believes, are yet to get their hiring processes right: “A lot of the measures that we use to sort of evaluate employees are fictional – when you say years of experience, it has no correlation to employee outcomes or the quality of employee you get long term. It’s the same thing as college degree, there’s no correlation.” So when Vin is trying to build a highly specialised team, what does he do? “We have to teach specialisation, we can’t expect it. We can’t bring someone in and call them a Data Scientist and hope that they train up. You end up with teams that are exactly the same because they have hired the same people, people who reinforce the bias of what they do, and that is where true leadership needs to come in.” A specialised team made up of individuals who bring their own ideas to the table is more important than ever, particularly as businesses demand more from their Data teams. Gone are the days of one-size-fits-all models. Businesses now want something tailored to them: “Custom models are huge. The “import from…” Machine Learning development from three years ago adds value when it comes to wrangling and doing the Analysis, but when it comes to creating models companies are now expecting it to become a competitive advantage. Companies no longer want the same model that everyone else has, now it has to be differentiating.” These smart, customised models, he adds, will help businesses through the current pandemic. “The best models right now are adapting rather than reacting.”  However, he’s sceptical about the Data Science community becoming too preachy:  “When it comes to COVID-19 one message I want to send to the Machine Learning and Deep Learning community is ‘shut up’. We don’t have the Data! We have so many Data Scientists talking about something that’s very important to get right. If you get it wrong the consequences and the credibility we will lose as a field is enormous.” Indeed, discussions about the lack of quality Data on COVID-19 are widespread at the moment and raise concerns for Vin: “What the last two and a half months has revealed is the danger of bad Data, the danger of assumptions that are hidden in Data that hasn’t been looked over well or wasn’t gathered well and was fed into these models that now aren’t robust. Of course, no model can account for something this drastic, but they should still be performing far better than they are right now.” Despite these concerns, Vin believes any change in the world brings about opportunities for those in the Data and technology space. “What I’ve been trying to do ever since I joined the technology space is figure it out. It’s constantly evolving and it’s constantly changing. That’s really what has driven my journey. I’m always trying to figure out ‘what’s next’ over the next five years, ten years whatever it may be.” If you’re looking for your next Data Science, Machine Learning or Deep Learning role, or want to build out your own highly-specialised team, we may be able to help.  Take a look at our latest opportunities or get in touch with one of our expert consultants to find out more.   

RELATED Jobs

Salary

£40000 - £60000 per annum + Shares

Location

London

Description

join a growing start-up, build a brand new product from scratch and help deliver the product not only in the UK, but internationally!

Salary

US$160000 - US$180000 per year

Location

New York

Description

Are you looking to join rapidly growing tech business where you can build an engineering team from scratch?

Salary

US$200000 - US$240000 per annum + Benefits

Location

Boston, Massachusetts

Description

A Global Healthtech Company with a life-changing mission statement are looking for a Senior Director Level Enterprise Architect to join them!

recently viewed jobs