Data engineers, the unsung heroes of data science

our consultant managing the role
Posting date:5/21/2018 6:33 PM
Before you can build a house, you need a blueprint of its design and schematics. When you begin construction, you must first lay the foundation upon which it will be built. Tangible products taken step-by-step to create first a house and then a home. However, in the world of data science, companies seem to have skipped the blueprints and foundational aspects and gone straight for the aesthetics. But, how do you decorate a house before it’s built?

A house without a foundation becomes a house of cards and the same is true of data analysis. Before the data scientists can process and analyse data, first must come the engineers. The Data Engineers who lay the digital foundation and set the parameters, who create the data lakes and platforms, so the data analysts have something to make sense of. As high as the demand is for data scientists, the demand and the need, is even greater for data engineers, yet a shortage remains.


Where are the Data Engineers?

Data engineering jobs outnumber data scientist jobs nearly four to one according to a quick search on job boards such as Glassdoor and Indeed. Yet, the complex technical nature of data engineering to support data scientists takes more than a degreed education. Unlike data analysts, data scientists, and other data professionals who can land a mid-level job directly out of university, data engineers cannot.

Ultimately, it takes between five to ten years for mid-level data engineers to gain enough experience for practical application. As such, systems do not yet exist in schools and universities to supplement data engineers undergraduate or postgraduate degrees in preparation for real life work experience in the field. However, once the experience is gained, it can take a company who has hired a data engineer up to two years to catch up with its competition.

With the pace of change in the tech world, this can be detrimental to both the business and the data science teams. Therein lies the Catch-22, data engineers must have experience before they can be hired, but there is no way to learn outside of hands-on, real life application.


Why You Need to Add a Data Engineer to Your Data Science Team

A data science team is not complete without a data engineer. Why? Because just like building a house, grand schemes and ideas to solve complex business problems, must first have a foundation. Data engineers are that foundational support of experts who design, build, and maintain data-based systems and organizational operations.

Not only do data engineers lay the foundation upon which data can be built, analysed, and ultimately translated to business professionals, it must also be timely.  Timely data leads to more data and better predictions.

Data engineers are not completely siloed from data science teams, they are also responsible for deploying the code and models that are written by data scientists. For more on the reasons data engineering is more important than data science for companies today, check out this article from Captech Consulting.

Data Science Team Seeks Data Engineer

Companies know data drives business and they know the importance of data professionals. However, they may mistakenly assume either that their data teams can pick up engineering experience as they work their way through a project or they simply assume the titles are interchangeable.

In the world of data engineering, there is no entry level job. Experience trumps education in this field.

Like the once siloed data science team now integrated across the business with sales, marketing, and advertising departments, so must the role of data engineer be integrated. This is not a marriage of convenience, but of necessity in order to stay ahead of the competition. Together, your fully integrated data teams – data engineering and data science now on equal footing - will be able to help your business reach better predictions faster, making you a voice of authority in your discipline.

Your Turn: Route to the Role of Data Engineer

The route to the role of Data Engineer may seem daunting with the catch-22 that experience supersedes education. So, in the spirit of collaboration, we thought we’d ask for your thoughts and opinions on a few items of interest such as how we can educate aspiring data engineers and get them into companies faster. What kind of cross-training programs might businesses and schools employ to fill the shortage? What other backgrounds are we overlooking as businesses seek to find and engage this most critical role within their data science teams?

According to the website Datanami, 2018 will be the year of the data engineer. If this is you, then we may have a role for you.

Related blog & news

With over 10 years experience working solely in the Data & Analytics sector our consultants are able to offer detailed insights into the industry.

Visit our Blogs & News portal or check out the related posts below.

Why A Good Work-Life Balance Is Better for Business

Contrary to American sitcoms, work life balance isn’t about sitting in coffee shops contemplating life and complaining about work. However, there are plenty of jobs where you can work from or in a coffee shop. The rise of virtual, remote, and contractual roles has contributed to the demand for work life balance. But, sometimes, in our tech-led world, where business can follow us anywhere, the balance becomes more about setting boundaries. It’s about putting down our mobile phones, closing our laptops, and dipping our toes into other waters.  Where Does Your Country Fit on the Work-Life Balance Scale? European countries have been leading the way with work-life balance for some time, with the Netherlands topping the list at number one. With the UK sitting at number 29 out of the 38 countries in the Organisation for Economic Co-operation and Development (OECD), what’s tipping the scales? 13% of British employees work 50 or more hours per week versus 0.5% of people in the Netherlands work those long hours. The average Brit is therefore only setting aside 14.9 hours for leisure and personal care (including eating and sleeping) a day versus those in the Netherlands who dedicate 15.9 hours. Countries in the Nordics work a maximum of 48-hours per week. However, the reality is significantly lower, with the Finnish working an average of 36.2 hours a week, the Swedes 35.9 hours, Norwegians at 34 hours, and the Danes just 32 hours.Denmark, Finland, Sweden, Norway, and Iceland have become renowned for fostering optimal work-life balance. But, though the Netherlands sits at the number one spot on the OECD, the Danes top the list as the happiest in the world. The Danish welfare model, characterised by quality of life and a good work-life balance offers: Flexible working conditions and social support networks, including maternity leave and childcare facilities. A high degree of flexibility at work – often including adaptable start times and the ability to work from home. Lunch breaks are often at a designated time each day, enabling colleagues to interact, eat together, and get away from their desks. There is a minimum 5 weeks’ paid holiday for all wage earners. The Danish welfare society is characterised by quality of life and a good work-life balance. Work-life balance for the Danes is a healthy balance of priorities. As important as career and ambition is, are is just as important to balance life outside work (pleasure, leisure, family, and health). This understanding of balance not only puts Denmark at the top of the international equality table, it also contributes to a generally high standard of living. Further research shows 33% of working American adults work over the weekend and on holidays. This, in turn, has led 66% to say they don’t feel they have a good work-life balance. One of the main drivers contributing to the need to always be “on” and available is 24/7 technology.  For example, if an employer emails, texts, or rings an employee at dinnertime, the employee often feels compelled to answer straightaway. While 57% of those surveyed feel technology has ruined the family dinner, 40% believe it is okay to answer an urgent call or email at the dinner table. So, it comes back to boundaries and not feeling guilty about ‘switching off’ for a few hours or a few days to ‘recharge’. What Companies are Doing to Improve Work-Life Balance  Nordic businesses remain at the top of the list for best work-life balance. Though much of it is dictated by strict Nordic Labour laws, companies outside the Nordics are beginning to take pages from their playbook.  At a business in Helsinki, Finland, employees are encouraged to go home on time at the end of their day. Often this falls around 5:00pm, though leaving earlier to say, go to a child’s sports activity, is always a guilt-free option.  Like many European businesses, employees also receive five weeks of paid vacation each year. Everyone gets stock options and teams are small with the ability to make autonomous decisions. The theory: this team is closest to the project, they know what is best for it. No management approval required, but only to help share in lessons learned. Many Nordic businesses have shortened hours and a focus on family. By putting family first, businesses report improved productivity and innovation, less absenteeism, and reductions in staff turnover. Other benefits can include: Ability to leave work 30-minutes early to pick up kids from school or take them to sports practice Ability to use sick days to take care of sick children Businesses regularly offer gym memberships, event discounts, leadership classes, and team-building exercises as well as opportunities for employees to take courses and further their education. At one business, in Sweden, for example, employees have access to a leisure centre and recreational activities such as fishing, tennis, and swimming. Though everyone has their own definition of what work-life balance means to them, it can be difficult to follow without government mandates, like in some European countries, or if you’re a small business. Our UK and Europe Salary Guide showed that, with over 98% of respondents working full time, at least some flexibility is now expected. We found that 53% of respondents work at home at least one day a week, and 56% have flexible working hours, highlighting that these ‘benefits’ are now becoming the norm.  Harnham Life As a business, we try to both reflect, and the lead the way with, developments that we see across the Data & Analytics industry. From ensuring our consultants leave on time two days per week to participate in pursuits outside work, to offering one fully-paid Charity Day per year, we place emphasis on creating an environment where our teams feel like they have a good work-life balance. By building a culture where a consultant can set up a book club or arrange a night out on the town, we have formed a business where employee welfare is prioritised.  Though everyone has their own definition of what work-life balance means to them, it can be difficult to follow without government mandates like in some European countries or if you’re a small business. The important thing is to do what’s right for you and sometimes turn off your phone, close your laptop, and meet up with some family or friends in that coffee shop.  Whether you’re looking for a permanent position with more benefits, or the freedom of a contract role, we’re here to help with your job search.  

Key Fraud Trends: How to Stay Safe in the Changing Fraudscape

Sharing and collecting data is part of our everyday lives. Whether our information is shared over social media, e-commerce sites, banks, or elsewhere, this can open up risks.  2017 saw the highest number of identity fraud cases ever, an increase in young people ‘money muling’ and higher bank account takeovers for over-60s. Whilst overall fraud incidences fell 6%, these cases highlight just some of the changing trends as fraud issues stem more from misuse than ever before. Dixons Carphone, Facebook and Ticketmaster are just three cases you may recognise from a string of high profile data breaches this year. Technological advances, more accessible and available data, coupled with an increased sophistication of fraud schemes, makes it more likely that data breaches and fraud attacks will become regular news items. But how is the fraud landscape changing and can technological advances be advantageous in detecting and reducing fraud? Identity fraud increasing for under 21s In June 2018, Dixons Carphone found an attack enabled unauthorised access to personal data from 1.2 million customers. It’s now been uncovered that the number is much higher, closer to ten times initial estimates. Whilst no financial information was directly accessed, personal data such as names, addresses and emails enable fraudsters to fake an identity. Younger fake identities are used more for product and asset purchases which typically require less stringent checks, such as mobile phone contracts and short-term loans.  In 2017, Cifas, a non-profit organisation working to reduce and prevent fraud and financial crime, reported the highest number of identity fraud cases ever. Under 21s are most at risk seeing a 30% increase as they engage more with online retail accounts. Whereas previously identity theft would manifest itself in fraudulent card and bank account activity, it’s now being used to make false insurance claims and asset conversion calling for stronger detection in these industries.  Young People Used as Money Mules This age group aren’t only being targeted for identity theft; there’s a 27% uplift in young people acting as money mules. ‘Money muling’ is a serious offence that carries a 14-year prison sentence in the UK. In most cases, younger people are recruited with the lure of large cash payments to facilitate movement of funds through their account, taking a cut as they go.  In a world where young lives are glamourised and luxurious goods are displayed over social media, this cut can be particularly appealing. Whether aware, believing the reward outweighs the risk, or unaware a money laundering crime is being committed, deeper fraud controls are needed across social media as much as bank accounts. This raises the question as to whether banks should be linking social media to customer details to stop money laundering early on? Increased bank account takeover for over 60s Cifas also reported an increase in account takeovers for over 60s for the same period. Seen by fraudsters as a less tech-savvy and therefore more susceptible demographic, over 60s are increasingly being targeted with online and social engineering scams. The same features which can make some over 60s a target for these scams, can also mean that account takeovers are not immediately noticed and reported, posing yet another difficulty for fraud monitoring and prevention. Vigilance and proactiveness is key. Here are three tips to get you started: Never give personal or security information to someone who contacts you out of the blue, either online, on the phone, or face to face. Always phone and check with the company first. If you make the call then you know you can trust the person on the other end. Check with your bank to see if they offer an elder fraud initiative such as a monitoring service that scans for suspicious activity and alerts customers and their families or educates seniors on types of scams and how to avoid them. When in doubt about something, delay and seek a second opinion. Check with your local library, government offices, or non-profit organisation for more top tips to stay safe from scams and social engineering.   Industry approach Traditionally, financial services organisations have been at the forefront of developing fraud controls; they are often the ones most impacted by the financial risk (the monetary cost of the attacks on their business) and regulatory risk (ensuring their business is adhering to regulations and controls). However, with modern day trends and the changing nature of fraud, all industries need to be focused on reputational risks and prevention. Single big events like Facebook and Dixon Carphone’s data breaches can have a far-reaching impact.  But, there is light at the end of the tunnel. Monzo, an online bank, which bills itself as the future of banking has stepped up the game when it comes to their customer’s security. Upon reports of fraudulent activity on customer cards, they took immediate action to correct the problem. Then they took things a step further, introducing digital analytics to help identify trends and patterns. As patterns emerged, Monzo then notified both the breached business and the authorities. Perhaps a cross-industry collaborative approach is needed as, after all, fraudsters are collaborating. By doing so, businesses will become more proactive, rather than reactive, and can put measures in place to stop potential fraud. If you’ve got a nose for numbers and want to help secure the reputation of businesses the world over, we may have a role for you.  To learn more, call our UK team at +44 020 8408 6070 or email us at ukinfo@harnham.com