Harnham
SEARCH
Shortlist

Senior Data Engineer

London
£500 - £600 per day

This vacancy has now expired. Please see similar roles below...

SENIOR DATA ENGINEER
LONDON BASED
£500-£600 per day
4-6 MONTHS

A chance to work as part of an agile team of Data Engineers building an AWS Data Platform from scratch using Scala/Spark!

THE COMPANY:

As a Senior Data Engineer you will be working for a leading client in the insurance industry. You will be part of a team of around 16 Data Engineers and 16 Data Scientists, BI Developers and DevOps Engineers in helping to set up a sophisticated Data platform from scratch.

THE ROLE:

As a Senior Data Engineer you will be heavily involved in building ETL pipelines using Scala/Spark so must be very proficient with this technology. You will be working in an AWS environment so familiarity with many of its key services is extremely useful. As a Senior Data Engineer you may also be helping to introduce streaming technologies such as Kafka or Kinesis for building a clients architecture so any experience with either of these technologies is valuable. As a Senior Data Engineer you will also be using Databricks and tools such as Airflow for scheduling so commercial use of these would also be helpful.

YOUR SKILLS AND EXPERIENCE:

  • Extensive experience with Scala, specifically Scala in Spark
  • Good working knowledge of AWS and services such as S3, Lambdas, Glue
  • Worked with streaming technologies such as Kafka/ Kinesis
  • Good experience with technologies such as Databricks/Airflow
  • Experience of technologies such as Kubernetes/ Docker

HOW TO APPLY:
Please register your interest by sending your CV to Anna Greenhill via the Apply link on this page.

Send similar jobs by email
62459/AG
London
£500 - £600 per day
  1. Contract
  2. Big Data

Similar Jobs

Data Engineer
Salary

£40000 - £50000 per annum

Location

Birmingham, West Midlands

Description

Exciting Data Engineering role where you will have full autonomy across multiple projects and products!

ADD TO SHORTLIST
Senior Manager - Data & Cloud
Salary

£70000 - £80000 per annum

Location

Birmingham, West Midlands

Description

Up to £80,000 + Competitive bonus & benefits

ADD TO SHORTLIST
Network Security Engineer - (Azure)
Salary

£70000 - £90000 per annum

Location

London

Description

Exciting new opportunity to join an exciting Fintech!

ADD TO SHORTLIST
Senior Data Engineer
Salary

£70000 - £90000 per annum

Location

London

Description

Exciting Data Engineering role where you will have full autonomy across multiple projects and products!

ADD TO SHORTLIST

Harnham blog & news

With over 10 years experience working solely in the Data & Analytics sector our consultants are able to offer detailed insights into the industry.

Visit our Blogs & News portal or check out our recent posts below.

How Are Data & Analytics Professionals Mapping COVID Trends With Data?

The coronavirus pandemic has impacted industries across the globe. There’s no ignoring that simple fact. This disruption (most notably) caused devastating effects in two strands: to our health and to business operations. As the virus spread, the health and wellbeing of people in society worsened, and businesses felt the strain of projects being placed on hold, and work slowing or completely grinding to a halt. As of the 24th February 2021, the disease has infected more than 112,237,188 people, with 2,487,349 reported deaths. For Data & Analytics professionals, it soon became evident that they could use their skills to help. Using the mass of data available, professionals and researchers turned to big data analytics tools to track and monitor the virus’s spread, along with a variety of trends. Here’s how: Genomics and sequencing Life science is a significant application within Data & Analytics and explores the study of all living things on earth. One particular section of this study looks at the concept of genomic sequencing.  Genomic sequencing is significant as it allows us looks at the entire genetic code of a virus – in this case, COVID-19. Most importantly, the technique means that researchers and analysts can identify dangerous mutations and track movements of specific variants. We know that the UK has the most advanced system for tracing covid variants too. Last year, Britain launched one of the world’s largest coronavirus sequencing projects, by investing £20 million in the Covid-19 Genomics UK consortium. In a group that included NHS researchers, public health agencies, academic partners and the Wellcome Sanger Institute, they set out to map the genetic code of as many strains of the coronavirus as possible. And the buy-in paid off. It took the US approximately 72 days to process and share each genetic sequence, compared with 23 days for UK researchers, according to figures compiled by the Broad Institute with data from Gisaid. Tech giants stepping in Ultimately, your organisation is more agile than you think it is. Regardless of the size of the business, or the industry in which it operates, the sector’s response in applying analysis and data to track the coronavirus was nothing short of miraculous. Google introduced a series of features such as popular times and live busyness, COVID-19 alerts in transit, and COVID checkpoints in driving navigation in order to keep their one billion (and growing) app users safe. They also introduced the COVID layer in Maps, a tool that shows critical information about COVID-19 cases in a given area, allowing their customers  to make informed decisions about where to go and what to do. Apple also released a mobility data trends tool from Apple Maps. This data was shared in order to provide insights to local governments and health authorities so that they could support mapping specific covid trends. These first-hand examples indicate the influence and power of using data to better our understanding of the virus. Before the coronavirus pandemic, professionals, businesses and industries alike worked in siloes. What we have witnessed since has been very much the opposite, as experts quickly came together to begin mapping out data requirements and supporting the world’s focus to improve the public’s health and get businesses back on their feet. Without Data & Analytics, none of this would be possible. If you're looking to take the next step in your career or build out a diverse Data & Analytics team, we may be able to help. Take a look at our latest opportunities or get in touch with one of our expert consultants to find out more. 

Amy Topcakan

Amy Topcakan

25. February 2021

News

Using Data Ethically To Guide Digital Transformation

Over the past few years, the uptick in the number of companies putting more budget behind digital transformation has been significant. However, since the start of 2020 and the outbreak of the coronavirus pandemic, this number has accelerated on an unprecedented scale. Companies have been forced to re-evaluate  their systems and services to make them more efficient, effective and financially viable in order to stay competitive in this time of crisis. These changes help to support internal operational agility and learn about customers' needs and wants to create a much more personalised customer experience.  However, despite the vast amount of good these systems can do for companies' offerings, a lot of them, such as AI and machine learning, are inherently data driven. Therefore, these systems run a high risk of breaching ethical conducts, such as privacy and security leaks or serious issues with bias, if not created, developed and managed properly.  So, what can businesses do to ensure their digital transformation efforts are implemented in the most ethical way possible? Implement ways to reduce bias From Twitter opting to show a white person in a photo instead of a black person, soap dispensers not recognising black hands and women being perpetually rejected for financial loans; digital transformation tools, such as AI, have proven over the years to be inherently biased.  Of course, a computer cannot be decisive about gender or race, this problem of inequality from computer algorithms stems from the humans behind the screen. Despite the advancements made with Diversity and Inclusion efforts across all industries, Data & Analytics is still a predominantly white and male industry. Only 22 per cent of AI specialists are women, and an even lower number represent the BAME communities. Within Google, the world’s largest technology organisation, only 2.5 per cent of its employees are black, and a similar story can be seen at Facebook and Microsoft, where only 4 per cent of employees are black.  So, where our systems are being run by a group of people who are not representative of our diverse society, it should come as no surprise that our machines and algorithms are not representative either.  For businesses looking to implement AI and machine learning into their digital transformation moving forward, it is important you do so in a way that is truly reflective of a fair society. This can be achieved by encouraging a more diverse hiring process when looking for developers of AI systems, implementing fairness tests and always keeping your end user in mind, considering how the workings of your system may affect them.  Transparency Capturing Data is crucial for businesses when they are looking to implement or update digital transformation tools. Not only can this data show them the best ways to service customers’ needs and wants, but it can also show them where there are potential holes and issues in their current business models.  However, due to many mismanagements in past cases, such as Cambridge Analytica, customers have become increasingly worried about sharing their data with businesses in fear of personal data, such as credit card details or home addresses, being leaked. In 2018, Europe devised a new law known as the General Data Protection Regulation, or GDPR, to help minimise the risk of data breaches. Nevertheless, this still hasn’t stopped all businesses from collecting or sharing data illegally, which in turn, has damaged the trustworthiness of even the most law-abiding businesses who need to collect relevant consumer data.  Transparency is key to successful data collection for digital transformation. Your priority should be to always think about the end user and the impact poorly managed data may have on them. Explain methods for data collection clearly, ensure you can provide a clear end-to-end map of how their data is being used and always follow the law in order to keep your consumers, current and potential, safe from harm.  Make sure there is a process for accountability  Digital tools are usually brought in to replace a human being with qualifications and a wealth of experience. If this human being were to make a mistake in their line of work, then they would be held accountable and appropriate action would be taken. This process would then restore trust between business and consumer and things would carry on as usual.  But what happens if a machine makes an error, who is accountable?  Unfortunately, it has been the case that businesses choose to implement digital transformation tools in order to avoid corporate responsibility. This attitude will only cause, potentially lethal, harm to a business's reputation.  If you choose to implement digital tools, ensure you have a valid process for accountability which creates trust between yourself and your consumers and is representative of and fair to every group in society you’re potentially addressing.  Businesses must be aware of the potential ethical risks that come with badly managed digital transformation and the effects this may have on their brands reputation. Before implementing any technology, ensure you can, and will, do so in a transparent, trustworthy, fair, representative and law-abiding way.  If you’re in the world of Data & Analytics and looking to take a step up or find the next member of your team, we can help. Take a look at our latest opportunities or get in touch with one of our expert consultants to find out more.

Henry Rodrigues

Henry Rodrigues

21. January 2021

News

Recently Viewed jobs