Harnham
SEARCH
Shortlist

Senior Data Engineer

London
£65000 - £85000 per annum + Additional Benefits
Apply Save job

Senior Data Engineer

London

£85,000 + Bonus + Benefits

The Company

Join one of the largest travel companies worldwide, in their fastest growing and most profitable team. You will work in an agile environment, focused on open source technologies working with partnerships across the globe, whilst maintaining personal development every step of the way.

The Role

In this role you will work on Bid Data and ensure that all signals are available in the data lake/warehouse and ensure that all analytics can be stored, enriched and processed for AI and Machine Learning. You will:

  • Work using Python, Scala and Java in equal measure across multiple projects
  • Work in a DevOps environment - if you can build it you can run it in the AWS platform.
  • Implement the tools and processes to handle performance, scale, availability and accuracy

Key Skills and Requirements

  • Experienced in Java, Scala and/or Python in the cloud
  • Understand AWS ecosystems
  • Background with a variety of technologies e.g. Hadoop, Spark, Hive, Presto

HOW TO APPLY

Interested? Please register your interest by submitting your CV directly by applying to this advert.

Apply Save job
Send similar jobs by email
VAC - 40381
London
£65000 - £85000 per annum + Additional Benefits
  1. Permanent
  2. Big Data

Similar Jobs

Data Engineer
Salary

£40000 - £50000 per annum

Location

Birmingham, West Midlands

Description

Exciting Data Engineering role where you will have full autonomy across multiple projects and products!

ADD TO SHORTLIST
Network Security Engineer - (Azure)
Salary

£70000 - £90000 per annum

Location

London

Description

Exciting new opportunity to join an exciting Fintech!

ADD TO SHORTLIST
Senior Data Engineer
Salary

£70000 - £90000 per annum

Location

London

Description

Exciting Data Engineering role where you will have full autonomy across multiple projects and products!

ADD TO SHORTLIST
Principal Software Engineer
Salary

US$170000 - US$180000 per annum

Location

New York

Description

Do you want to join an artificial intelligence company that has designed market-disrupting applications for the better good?

ADD TO SHORTLIST

Harnham blog & news

With over 10 years experience working solely in the Data & Analytics sector our consultants are able to offer detailed insights into the industry.

Visit our Blogs & News portal or check out our recent posts below.

Using Data Ethically To Guide Digital Transformation

Over the past few years, the uptick in the number of companies putting more budget behind digital transformation has been significant. However, since the start of 2020 and the outbreak of the coronavirus pandemic, this number has accelerated on an unprecedented scale. Companies have been forced to re-evaluate  their systems and services to make them more efficient, effective and financially viable in order to stay competitive in this time of crisis. These changes help to support internal operational agility and learn about customers' needs and wants to create a much more personalised customer experience.  However, despite the vast amount of good these systems can do for companies' offerings, a lot of them, such as AI and machine learning, are inherently data driven. Therefore, these systems run a high risk of breaching ethical conducts, such as privacy and security leaks or serious issues with bias, if not created, developed and managed properly.  So, what can businesses do to ensure their digital transformation efforts are implemented in the most ethical way possible? Implement ways to reduce bias From Twitter opting to show a white person in a photo instead of a black person, soap dispensers not recognising black hands and women being perpetually rejected for financial loans; digital transformation tools, such as AI, have proven over the years to be inherently biased.  Of course, a computer cannot be decisive about gender or race, this problem of inequality from computer algorithms stems from the humans behind the screen. Despite the advancements made with Diversity and Inclusion efforts across all industries, Data & Analytics is still a predominantly white and male industry. Only 22 per cent of AI specialists are women, and an even lower number represent the BAME communities. Within Google, the world’s largest technology organisation, only 2.5 per cent of its employees are black, and a similar story can be seen at Facebook and Microsoft, where only 4 per cent of employees are black.  So, where our systems are being run by a group of people who are not representative of our diverse society, it should come as no surprise that our machines and algorithms are not representative either.  For businesses looking to implement AI and machine learning into their digital transformation moving forward, it is important you do so in a way that is truly reflective of a fair society. This can be achieved by encouraging a more diverse hiring process when looking for developers of AI systems, implementing fairness tests and always keeping your end user in mind, considering how the workings of your system may affect them.  Transparency Capturing Data is crucial for businesses when they are looking to implement or update digital transformation tools. Not only can this data show them the best ways to service customers’ needs and wants, but it can also show them where there are potential holes and issues in their current business models.  However, due to many mismanagements in past cases, such as Cambridge Analytica, customers have become increasingly worried about sharing their data with businesses in fear of personal data, such as credit card details or home addresses, being leaked. In 2018, Europe devised a new law known as the General Data Protection Regulation, or GDPR, to help minimise the risk of data breaches. Nevertheless, this still hasn’t stopped all businesses from collecting or sharing data illegally, which in turn, has damaged the trustworthiness of even the most law-abiding businesses who need to collect relevant consumer data.  Transparency is key to successful data collection for digital transformation. Your priority should be to always think about the end user and the impact poorly managed data may have on them. Explain methods for data collection clearly, ensure you can provide a clear end-to-end map of how their data is being used and always follow the law in order to keep your consumers, current and potential, safe from harm.  Make sure there is a process for accountability  Digital tools are usually brought in to replace a human being with qualifications and a wealth of experience. If this human being were to make a mistake in their line of work, then they would be held accountable and appropriate action would be taken. This process would then restore trust between business and consumer and things would carry on as usual.  But what happens if a machine makes an error, who is accountable?  Unfortunately, it has been the case that businesses choose to implement digital transformation tools in order to avoid corporate responsibility. This attitude will only cause, potentially lethal, harm to a business's reputation.  If you choose to implement digital tools, ensure you have a valid process for accountability which creates trust between yourself and your consumers and is representative of and fair to every group in society you’re potentially addressing.  Businesses must be aware of the potential ethical risks that come with badly managed digital transformation and the effects this may have on their brands reputation. Before implementing any technology, ensure you can, and will, do so in a transparent, trustworthy, fair, representative and law-abiding way.  If you’re in the world of Data & Analytics and looking to take a step up or find the next member of your team, we can help. Take a look at our latest opportunities or get in touch with one of our expert consultants to find out more.

Henry Rodrigues

Henry Rodrigues

21. January 2021

News

The Dialogue: Keeping Data Secure While Working Flexibly

Last week, Peter Schroeter and Ryan Collins, head of DevSecOps at Upvest, and co-founder of RapidGroup, discussed how to keep Data secure while working flexibly.  Ryan brings to the table 12 years of experience from Server Administrator to CTO roles as well as experience as a Contractor. And as a business owner himself, he sees where the shortage falls and perhaps a way to fill the gap. Security is in the Spotlight Security is a priority for many businesses today. Avoiding negative PR has caused an internal shift in which companies take more care with their Data. There is also a Catch-22. In order to provide higher levels of security, businesses are slowing down their developers and software engineers, and taking more engineering time which in turn costs more money. Security before cost is becoming the new reality. The contradiction of deployment, project run time, and budgets are only part of the bigger picture. Compromise is key and follow these three tenets. Don’t leave anything open to vulnerability.Focus on auditability.Offer more training for Software Engineers. The Security-Focused Skills Gap How can you push security forward in a meaningful way when you can’t find the people you need with relevant experience? There’s a big gap in the market right now for security-oriented DevOps engineers. The skillsets many businesses are looking for include: Google Platform AWS and possibly Azure with a modern suite of tools with Teraform.Ability to float to the development side and work in SRE to ensure things are stable and scalable.DevOps Engineers with experience in the GO stack are especially hard to find. Companies want to go with what they know. However, there is a shift toward a more remote-friendly and contractor culture. When you can’t find the talent you need, sometimes it’s best to bite the bullet, and consider a Contractor. The Case to Hire a Contractor If this year has taught us anything, it’s that we don’t have to be confined to an office or to even one location. And yes, while there is an element of risk to being a Contractor, there are benefits to both sides. Contractors are compensated higher because businesses have lower HR costs, less tax regulations, payroll, and reporting to do.  Though there is some risk, there is always work for because so many companies want to secure their data. There’s always something to be optimised, always something which needs attention. You won’t be without work for long, if you have the skills. Disconnect Between Capability and Desire in the Market Some candidates have mentioned they have 60% of the skills required, but not enough project-based experience. How do you reconcile the disconnect?  It’s hard to specialise in DevOps, DevSecOps, and similar roles because it’s about automation, you have to be that "Swiss Army Knife", you have to live in the middle. You have to know how to get into the code, you have to know how to get in and do the CI workflows, etc. It’s almost Developer plus value-added skills or security engineer plus value-added skills. Remote Working Habits You Need Now Have a separated space set up for work.If you have a mix of people in office and those working remote, you have to make sure they’re communicating with each other.Rolling coffee breaks within Google Meet or something similar.Have a task management and time tracking system used by employees both on and offline.Build culture by picking at random two people and putting them together for a half hour to have a conversation that isn’t about work.  Startup vs Legacy Hiring Don’t box yourself in, but understand startups have a unique set of skills they need and most often without the budget to train someone. Whereas legacy businesses more often than not have the budget to train someone in the skillset they need. However, it’s important to note, the tech stack itself doesn’t really change. Best Practices for Async-Comm Teams Remember not everyone works in the same time zone. Don’t expect and immediate answer, and if you need an immediate answer, pick up the phone. Make sure your team has remote tools such as Slack or Google Meet and is doing most of their communication this way, even if they are in an office. If you don’t, your remote workers could be missing key information. The Future of DevOps and Data Security Many businesses may see a shift toward a more open working environment which is a good balance for what works best for talent and is good for productivity as well. Ultimately, we’re all solving the same challenges What is most important when it comes to keeping Data secure? Most important is getting people, systems, and education in place to do something in the first place. In other words, build from concept rather than moving too fast and breaking too many things. How Can Prospective Candidates Prepare? Focus on continuous learning and getting your skillsets like automation tools. If you really want to get involved with security side and SRE, you really have to get involved with the development side, too. Using the modern tech stacks like the GOLANG, the RUST, the SWIFT on the mobile side, and there’s always new pieces to the puzzle.There’s room for all types of relationships. Whether you’re looking for a long-term role, a short-term role, or something in between, DevOps is a never-ending project, so continuous learning is key for both candidate and company. You can watch the full conversation below:

Ali Mirza

03. December 2020

News

Recently Viewed jobs