DATA ENGINEERING AND BIG DATA
TALENT SOLUTIONS
Processing Big Data has become crucial to countless businesses. Those who decide to pursue a career within Big Data face complex development challenges, too tough for relational database systems. After all, there is no Data Science without Data Engineers.
Whilst businesses may have vast quantities of data at the ready, it holds no real value unless it can be stored, harnessed and utilised. It’s estimated that most UK companies have at least 100 terabytes of data stored which could be used for various purposes around the business, as well as newly accessible hard-to-process sources such as web data, image data, and social media data.
This is where those who work in Data Engineering become extremely valuable, as they develop Data Platforms that allow this kind of volume to be processed and used by Data Scientists and analysts across the business.
WHY
HARNHAM?
Our esteemed position as a global leader in data recruitment stems from our deep expertise in Big Data and Data Engineering, supported by hundreds of dedicated specialists located across the United States, Europe, and the United Kingdom.
Our extensive network enables us to deliver exceptional recruitment services, expertly matching our clients with top-tier Data Engineering talent.
We acknowledge the distinct challenges and requirements of each organization's data infrastructure.
Therefore, our recruitment strategies are meticulously tailored, emphasizing a thorough understanding of and alignment with your specific business objectives and data technological needs.
OUR
SERVICES
-
Permanent and Contract Recruitment: We offer both permanent and contract recruitment solutions, adapting flexibly to support the dynamic requirements of your Big Data and Data Engineering projects.
-
Executive Search: Our executive search service focuses on identifying and securing top-tier leaders in the Data Engineering domain who can advance your strategic goals and technological advancements.
-
Industry-Specific Expertise: Our operations span multiple industries, providing specialized recruitment solutions tailored to the distinctive challenges and opportunities of your sector.
Contact us today to discover how our custom talent solutions can bolster your organization's Data Engineering prowess.
JOBS
LATEST
OPPORTUNITIES
Harnham are a specialist Data & AI recruitment business with teams that only focus on niche areas.
Lead Data Engineer
London
£80000 - £90000
+ Data Engineering
PermanentLondon
To Apply for this Job Click Here
Lead Data Engineer
Hybrid (London /Reading – 1-2 days per week)
Up to £90,000 + Benefits
Are you ready to take the lead in shaping world-class data infrastructure at one of the most influential analytics companies on the planet? We’re partnering with a global leader in marketing data and insights that’s seeking a Lead Data Engineer to help scale and evolve its cloud-based data ecosystem.
This is an opportunity to work on high-impact global datasets, enabling better analytics, stronger fraud prevention, and smarter decision-making across 90+ markets.
Why this role?
-
Play a key role in building and optimising global data pipelines supporting over 50 consumer panels worldwide.
-
Lead the technical strategy for data architecture, quality, and scalability.
-
Collaborate closely with data scientists, analysts, and software engineers to drive innovation in data engineering.
-
Hybrid flexibility: 1-2 days per week in London, Reading, or Southbank office, balance remotely.
-
Join a global organisation with the resources of a world leader and the agility of a fast-moving data tech team.
What you’ll be doing:
-
Leading the design, development, and maintenance of data pipelines and infrastructure across AWS and Azure.
-
Building and evolving systems that detect and prevent fraud and data anomalies in large-scale survey data.
-
Integrating diverse data sources (APIs, databases, external datasets) into a unified analytics ecosystem.
-
Automating data ingestion and transformation workflows using modern ELT/ETL best practices.
-
Implementing monitoring and alerting systems to ensure high data quality and reliability.
-
Mentoring a small team of data engineers, driving excellence and continuous learning.
-
Partnering with Data Science and BI teams to deliver reliable, low-latency data for analytics and reporting.
What we’re looking for:
-
5+ years’ experience in data engineering, including technical or team leadership.
-
Expertise in AWS (preferred) or Azure, and experience managing data pipelines in the cloud.
-
Strong skills in Python and SQL, with a solid grasp of data warehousing concepts.
-
Experience with DBT, Redshift, or Postgres, and familiarity with BI tools (Power BI preferred).
-
Proven understanding of CI/CD principles, DevOps, and infrastructure automation.
-
Excellent communication skills – able to collaborate across technical and business functions.
-
A passion for data-driven problem solving, innovation, and continuous improvement.
✨ Nice-to-haves:
-
Experience with data orchestration tools (Airflow, Prefect).
-
Knowledge of data observability or cataloguing tools (Monte Carlo, OpenMetadata).
-
Familiarity with large-scale consumer data or survey environments.
To Apply for this Job Click Here
Lead Data Engineer
London
£80000 - £90000
+ Data Engineering
PermanentLondon
To Apply for this Job Click Here
LEAD DATA ENGINEER
£80,000 – £90,000 – LONDON
Company
Our client is a global leader in data and analytics, working with major brands across 90+ markets. They combine deep industry expertise, vast datasets, and advanced technology to help businesses understand people and drive smarter decisions. The company is backed by a top-tier private equity firm and is currently investing heavily in its data and technology platforms, offering the feel of a scale-up within a global business.
Role
As a Lead Data Engineer, you’ll take ownership of designing, maintaining, and evolving the data pipelines that power large-scale analytical and machine learning models. You’ll work closely with Data Scientists, Analysts, and Software Engineers to deliver robust, scalable data infrastructure. The focus areas include building and optimising pipelines for survey data, improving anti-fraud models, and supporting new data science initiatives.
You’ll manage two mid-level engineers, providing technical direction, mentoring, and helping the team refine ways of working. The role is hands-on and collaborative, offering the chance to make tangible impact across data quality, scalability, and reliability.
Requirements
-
5+ years’ experience in Data Engineering with strong Python and SQL skills.
-
Background in building and maintaining ETL/ELT pipelines in a cloud environment (AWS or Azure).
-
Knowledge of data warehousing (e.g., Redshift, Postgres) and tools like DBT.
-
Familiarity with CI/CD, data modelling, and Agile practices.
-
Comfortable collaborating with Data Scientists and translating business needs into technical solutions.
-
Experience leading or mentoring other engineers.
Benefits
-
Hybrid working from either London or Reading.
-
Competitive salary and performance-based rewards.
-
The chance to work on high-volume, high-impact data systems used globally.
-
Strong internal development culture and access to senior technical mentors.
-
Join a business with ambitious growth plans and the backing of a global brand and major investors.
To Apply for this Job Click Here
Staff Analytics Engineer
$200000 - $250000
+ Data Engineering
PermanentCalifornia
To Apply for this Job Click Here
Staff Analytics Engineer
Location: Remote
Salary: $200-250k
Are you a data expert ready to drive transformative analytics at a cutting-edge cloud platform company? This fast-growing Series C innovator is revolutionizing how teams build, preview, and deploy high-performance digital applications.
About the Role
The Staff Analytics Engineer will lead the design and implementation of robust data models and pipelines, enabling business units to uncover insights and make data-driven decisions. This role is a unique chance to collaborate with engineering, product, and business teams, building scalable analytics systems that power operational efficiency and growth.
Key Responsibilities
- Design and maintain scalable data models and pipelines to serve as a single source of truth across the enterprise.
- Develop tools to enhance data accessibility, auditing, and validation, ensuring consistent, trustworthy datasets.
- Partner with cross-functional teams to uncover actionable insights and support decision-making.
- Build self-service analytics tools and dashboards for key stakeholders, streamlining workflows.
- Collaborate on advancing the company’s data platform using cutting-edge tools like dbt, Snowflake, and Superset.
- Lead initiatives to improve data quality, governance, and discoverability, ensuring alignment with business goals.
About You
- 8+ years of experience in analytics engineering or a similar role in a modern tech company.
- Advanced proficiency in SQL, with experience working on complex datasets.
- Expertise in data platforms like Snowflake or BigQuery and transformation tools like dbt.
- Familiarity with programming languages such as Python, Java, or Go.
- Experience with BI tools like Hashboard or Superset, and a strong grasp of schema design and dimensional modeling.
- Strong communication skills and the ability to collaborate in a fast-paced environment.
Be part of a forward-thinking team where data is at the core of every decision. If you’re passionate about transforming data into meaningful insights, this role offers the chance to work on impactful projects, collaborate with top talent, and make a lasting difference.
To Apply for this Job Click Here
Principal Software Engineer – Data Platform
San Francisco
$230000 - $280000
+ Data Engineering
PermanentSan Francisco, California
To Apply for this Job Click Here
Principal Software Engineer – Data Platform
San Francisco, CA (Remote Eligible – US Only)
$230K-$280K base + equity
Are you ready to lead the design and build of a world-class data platform from the ground up?
A high-growth, product-led tech company is looking for a Principal Software Engineer to lead the build-out of a modern, scalable data platform from the ground up. If you’re excited about solving massive data challenges and shaping the infrastructure that supports real-time analytics and AI/ML at scale, this role is for you.
As the most senior technical IC on the Data Platform team, you’ll partner with leadership across Engineering, Product, Security, and Compliance to design a low-latency, cloud-native data architecture using best-in-class tools like Kafka, Iceberg, ClickHouse, Tinybird, and Snowflake.
Why this role?
- Join a growing Data Platform team
- Own the technical vision and architecture end-to-end
- Tackle petabyte-scale data challenges and real-time streaming use cases
- Work with a cutting-edge tech stack (Kafka, Iceberg, ClickHouse, Snowflake, etc.)
- Drive platform evolution to support AI/ML and company-wide analytics
What You’ll Do
- Design and build a next-gen data platform to power product and internal use cases
- Architect for batch + streaming, scalability, fault tolerance, and low latency
- Lead data modeling, warehousing, governance, and ingestion best practices
- Champion engineering excellence and mentor senior engineers
- Align technical decisions with long-term business strategy
- Optimize for performance, cost, and resiliency in a modern cloud environment
What We’re Looking For
- 8+ years of experience in data engineering or platform architecture
- 5+ years at the Principal Engineer level or equivalent
- Deep expertise with Kafka (and its ecosystem), streaming data systems
- Strong experience with ClickHouse, Iceberg, Snowflake, or similar tools
- Cloud-native architecture expertise (AWS, GCP, or Azure)
- Track record of designing data systems for compliance and governance
- Experience scaling systems to support petabyte-level workloads
- Excellent communication, collaboration, and leadership skills
- Bonus: Experience enabling AI/ML platforms or preparing for IPO-scale growth
To Apply for this Job Click Here
Director of Data Engineering
New York
$300000 - $350000
+ Data Engineering
PermanentNew York
To Apply for this Job Click Here
Director of Data Engineering
Location: San Francisco or New York (Hybrid)
Compensation: $300-350K base + Equity
A $3B+ Series E SaaS company at the forefront of AI-powered enterprise transformation is looking for a Director of Data Engineering to scale its Data Platform and Analytics Engineering functions. This is a rare opportunity to lead the next phase of data infrastructure, powering everything from LLM-based product experiences to executive analytics and real-time experimentation.
What You’ll Lead:
- Data Platform Strategy: Define the roadmap for scalable, cloud-native infrastructure-streaming, compute, and storage-including technologies like Apache Iceberg, Kafka, and Snowflake.
- Team & Technical Leadership: Grow and mentor a high-performing data engineering team (2 → 10+), setting engineering standards and fostering technical excellence.
- Cross-Functional Collaboration: Partner with ML, Security, Legal, and Product teams to enable AI/ML platforms, telemetry pipelines, and governance at scale.
- Systems at Scale: Architect robust, real-time pipelines and petabyte-scale infrastructure with streaming pre-aggregations, observability, and cost efficiency.
- Operational Maturity: Drive platform reliability and compliance-readiness as the company prepares for IPO-implementing best practices in governance, testing, and infrastructure as code.
What We’re Looking For
- 10+ years in data engineering, with 5+ years of engineering leadership
- Deep experience scaling data platforms in high-growth SaaS, infrastructure or AI-first environments
- Expert in Kafka, Apache Iceberg, Snowflake, dbt, and cloud infrastructure (AWS or GCP)
- Proven success supporting ML/AI platforms and structured/unstructured data needs
- Strong track record of partnering with legal and compliance to build IPO-ready systems
- A collaborative, technical leader who thrives in fast-paced, high-impact settings
Why This Role
- Lead the data engineering charter at one of the most innovative companies in enterprise AI
- Own systems that power product analytics, experimentation, personalization, and LLM workloads
- Work side-by-side with the C-suite and play a pivotal role in company-scale transformation
To Apply for this Job Click Here
Staff Data Platform Engineer
San Francisco
$200000 - $245000
+ Data Engineering
PermanentSan Francisco, California
To Apply for this Job Click Here
Staff Data Platform Engineer
US Remote
$200-245k base + Equity
We’re working with a rapidly expanding consumer tech startup that recently raised $125M in Series D funding. As they continue to scale, they’re looking to hire a Staff Data Engineer to join their growing data team. If you have a strong foundation in software engineering, a collaborative mindset, and a passion for driving data strategy, this is the opportunity for you!
The Role:
As a Staff Data Engineer, you’ll be a crucial part of the team, ensuring data is accessible, reliable, and actionable across the company. Working alongside engineers, data scientists, and business teams, you’ll help build scalable data platforms that drive key business initiatives and enable data-driven decision-making.
What You’ll Do:
- Collaborate with engineering, data science, and business teams to understand data needs and implement effective solutions aligned with business objectives.
- Design and build scalable data infrastructure and pipelines to handle high volumes of data as the company grows.
- Lead the design and optimization of data architecture, ensuring it’s secure, high-performing, and scalable.
- Improve and streamline data pipelines to provide fast, reliable access to high-quality data.
- Contribute to the design of reporting frameworks and analytics tools that empower teams to make data-driven decisions.
- Work with security teams to ensure data privacy and compliance standards are met.
- Mentor junior engineers, share expertise, and promote best practices across the team.
- Stay updated on new technologies and suggest improvements to enhance processes and tool efficiency.
You’re a Great Fit If:
- You have 7+ years of experience in data engineering.
- You’ve worked at a leading consumer tech company known for engineering excellence.
- You’re an expert in SQL, Python, and data platform development.
- Nice-to-have: Familiarity with AWS, Kubernetes, Redshift, Athena, S3, and Airflow.
- You have hands-on experience with complex data infrastructures and optimizing data pipelines.
- You’ve led large-scale data projects that had a significant impact on your organization.
- You thrive in collaborative environments and have strong communication skills.
- You have an ownership mindset and take responsibility for delivering results.
To Apply for this Job Click Here
Director of Data Engineering
San Francisco
$300000 - $350000
+ Data Engineering
PermanentSan Francisco, California
To Apply for this Job Click Here
Director of Data Engineering
Location: San Francisco or New York (Hybrid)
Compensation: $300-350K base + Equity
A $3B+ Series E SaaS company at the forefront of AI-powered enterprise transformation is looking for a Director of Data Engineering to scale its Data Platform and Analytics Engineering functions. This is a rare opportunity to lead the next phase of data infrastructure, powering everything from LLM-based product experiences to executive analytics and real-time experimentation.
What You’ll Lead:
- Data Platform Strategy: Define the roadmap for scalable, cloud-native infrastructure-streaming, compute, and storage-including technologies like Apache Iceberg, Kafka, and Snowflake.
- Team & Technical Leadership: Grow and mentor a high-performing data engineering team (2 → 10+), setting engineering standards and fostering technical excellence.
- Cross-Functional Collaboration: Partner with ML, Security, Legal, and Product teams to enable AI/ML platforms, telemetry pipelines, and governance at scale.
- Systems at Scale: Architect robust, real-time pipelines and petabyte-scale infrastructure with streaming pre-aggregations, observability, and cost efficiency.
- Operational Maturity: Drive platform reliability and compliance-readiness as the company prepares for IPO-implementing best practices in governance, testing, and infrastructure as code.
What We’re Looking For
- 10+ years in data engineering, with 5+ years of engineering leadership
- Deep experience scaling data platforms in high-growth SaaS, infrastructure or AI-first environments
- Expert in Kafka, Apache Iceberg, Snowflake, dbt, and cloud infrastructure (AWS or GCP)
- Proven success supporting ML/AI platforms and structured/unstructured data needs
- Strong track record of partnering with legal and compliance to build IPO-ready systems
- A collaborative, technical leader who thrives in fast-paced, high-impact settings
Why This Role
- Lead the data engineering charter at one of the most innovative companies in enterprise AI
- Own systems that power product analytics, experimentation, personalization, and LLM workloads
- Work side-by-side with the C-suite and play a pivotal role in company-scale transformation
To Apply for this Job Click Here
Software Engineer – Python (Contract)
London
£700 - £800
+ Data Engineering
ContractLondon
To Apply for this Job Click Here
THE COMPANY
This organisation is pushing the boundaries of AI and data-driven software, developing next-generation tools that integrate advanced machine learning with scalable backend systems. They’re looking for an experienced Python Backend Engineer to support the build and integration of key AI components across their tech stack.
THE ROLE
As a Contract Python Backend Engineer, you’ll take ownership of backend development for a Python-based web application, ensuring smooth integration with AI/ML pipelines and data infrastructure. You’ll work closely with data engineers, AI specialists, and frontend developers to deliver robust, production-quality software.
Key Responsibilities:
-
Build and maintain Python web applications (FastAPI preferred)
-
Integrate AI/ML components with backend, data, and compute infrastructure
-
Deliver clean, well-documented code following best practices
-
Implement automated testing frameworks and CI/CD workflows
-
Monitor and improve performance metrics across deployed tools
-
Collaborate with technical teams to develop end-to-end data pipelines
YOUR SKILLS AND EXPERIENCE
Essential:
-
Proven Python backend development experience (FastAPI, Flask, or similar)
-
Strong cloud experience (GCP preferred) and understanding of core web app infrastructure
-
Skilled in automated testing (pytest) and modern DevOps tools (Git/GitHub, CI/CD)
-
Experience working in agile environments and delivering production-ready solutions
-
Degree in a computational, numerate, or life sciences discipline (or equivalent experience)
Preferred:
-
Docker and containerized application experience
-
Experience in scientific or pharmaceutical domains (e.g. genomics, proteomics)
-
Familiarity with AI/ML-powered applications and agent-based frameworks (LangGraph, PydanticAI, AutoGen)
-
Knowledge of AI/ML evaluation and iterative model improvement
-
Exposure to frontend technologies (React desirable)
THE BENEFITS
-
Competitive day rate (inside IR35)
-
Cutting-edge AI/ML project work
-
Flexible, collaborative environment with autonomy over technical decisions
To Apply for this Job Click Here
Senior Data Engineer
Manchester
£60000 - £70000
+ Data Engineering
PermanentManchester, Greater Manchester
To Apply for this Job Click Here
Role: Azure Senior Data Engineer
Location: Manchester, Lancashire
Hybrid working: go into office 1x day a week
Salary: £60,000 – £70,000 (dependent on experience)
Insight into the Company:
A large retail organisation – market leaders in their industry – are looking for a Senior Data Engineer to enter their team. You will be working in a small team of 3 (including you!) and they are looking to build up the team over the next year.
The ideal candidate will have experience with managing data warehouses from start to finish and liaise with IT teams regularly. The role will involve stakeholder management, working with diverse roles, such as software engineers, sales and marketing professionals and more. Therefore, communicating your projects needs and understanding others is an essential part of this role.
Role and Responsibilities:
- You will design, build and upgrade data pipelines
- You will work in CI/CD and with Software/ DevOps teams in the organisation
- You will have expertise in azure – from collecting, to transforming to loading!
Skills and Experience:
- Essential to have experience with:
- Azure
- SQL
- Regular stakeholder management
- Data warehouse management
- Power BI
- Desirable to have experience with:
- Bachelors in STEM subject – ideally computer science or engineering
- CI/CD methods
Interview Process:
- There are 3 stages to the process:
- Introductory conversation with the Head of Business Insights
- In person interview, focussing on techstacks
- Sign off conversation!
To Apply for this Job Click Here
CAN’T FIND THE RIGHT OPPORTUNITY?
GET IN TOUCH
today
If you can’t see what you’re looking for right now, send us your CV anyway – we’re always getting fresh new roles through the door.
Industry Hub
HARNHAM
News & Blog
With over 10 years experience working solely in the Data & AI sector our consultants are able to offer detailed insights into the industry.
Visit our Blogs & News portal or check out our recent posts below.
What’s Driving Northern UK’s Data & AI Hiring in 2025?
by Jamie Smith, Senior Manager at Harnham, UK. At a glance The Northern data and tech market…
The Key to a Strong AI Team: Diversity
by Tom Brammer, Senior Manager – AI and Machine Learning US Team Every company wants to build…
Why US Companies Are Scaling AI with Contractors in 2025
Artificial intelligence has arrived in the workplace and it’s changing how companies are structured, resourced, and scaled. …
How Sacher.AI Strengthened Its GenAI Delivery in Just 12 Days
by Thomas Savidge, Principal Consultant – AI, Machine Learning and Data Science Sacher.AI, a UK-based innovation lab…
From Zero to AI: How One Investment Firm Built a World-Class Team in 18 Days
A global investment advisor managing over $268 billion across real estate, equity, and credit needed to build…
How to Maximise Your Data’s Value: The Use of AI in Advanced Analytics
By Roshni Baillie, Recruitment Consultant – Marketing & Insight, Harnham One thing I’m hearing again and again…
AI Governance Starts with People
By Kiran Ramasamy, Managing Consultant, Harnham When it comes to AI it is no longer a question…
15,000 Placements Later: What We’ve Learned About the Global Data & AI Talent Market
By David Farmer, CEO, Harnham Global We haven’t just hit 15,000 placements, we’ve launched 15,000 careers,…
Download ‘How to Hire in AI’: Our AI Hiring How-to Guide
In a landscape dominated by discussion of AI, companies can find it hard to navigate the marketplace….
The Role of Data Governance in AI: Landscape in the Netherlands
By Ross Henderson & Robin Buitendijk, Harnham Netherlands The transformative power of technology has entered an exciting…
Testimonials
Client and
candidate
testimonials
A trusted partner of professionals
across the globe.
We understand the challenges our
customers face and offer the
recruitment solutions needed to drive
business success through Data &
AI.
Client – Director of People Operations
We’ve been collaborating with Harnham to recruit some key positions for our company, and they have been incredible partners. They team has been great and efficient. They understood our needs perfectly and provided us very talented candidates. We'll keep collaborating with them in the future. Thank you for your excellent job!
Client /
Director of People Operations
Client – Data Direction – Broadcast Media
I have worked with Harnham for over 2 years now as our main Analytics Recruiter partner and I am very pleased with the overall service and the people we have hired via Harnham. Harnham’s team is very good at understanding our recruitment needs and always come back with relevant and well scanned CVs, which saves me precious time.
Client /
Data Direction – Broadcast Media
Candidate – Senior Data Engineer
I've been a Senior Data Engineer for about 4-5 years and Harnham is one of the best recruitment agencies I worked with. There was a time when my offer fell through and the recruiter I was working with at Harnham jumped in to the rescue and scheduled interviews for me within a couple of days. Really good agency and some great people.
Candidate /
Senior Data Engineer
Candidate – Data Analyst
I’ve had a great experience with Harnham. The recruiter provided great advice and ensured that I understood everything, and even followed up with me after I started my new role to see how I was settling in. I would recommend Harnham as a great recruitment company for anyone looking for a role in data and analytics.
Candidate /
Data Analyst
Client – Director of Customer Data Science – Apparel
The team at Harnham have been outstanding in helping our company build a market-leading data science team. Harnham’s communication is great, and the recruiter’s take a consultative approach that’s really focused on provided value for both their clients and their candidates. They provide strong candidates that help us fill our roles with high achievers who are also a great fit for our culture.
Client /
Director of Customer Data Science – Apparel
Candidate – Market Research Analyst
Thanks to Harnham, I am now in a role that suits me. My recruiter was able to quickly identify the skills I had to offer and match me up with his client. It was a win-win for everyone. I was given great advice throughout the process, and I also appreciated the follow-ups that I received.
Candidate /
Market Research Analyst
Candidate – Chief Data Officer
It was a real treat to work with Harnham. My recruiter consistently went above and beyond to ensure that I navigated through the recruitment process as smooth as possible. I had been in contact with other professional at Harnham in the past and I can attest that the level of professionalism is always top-notch.
Candidate /
Chief Data Officer
Candidate – Digital Media and Analytics
I've recently got a short-term contract role through Harnham, and things couldn't have gone better! My recruiter was incredibly helpful throughout the entire process. They also provided incredible admin support so that I could focus on the actual job in hand, rather than having to spend too much time dealing with these matters.
