DATA ENGINEERING AND BIG DATA
TALENT SOLUTIONS
Processing Big Data has become crucial to countless businesses. Those who decide to pursue a career within Big Data face complex development challenges, too tough for relational database systems. After all, there is no Data Science without Data Engineers.
Whilst businesses may have vast quantities of data at the ready, it holds no real value unless it can be stored, harnessed and utilised. It’s estimated that most UK companies have at least 100 terabytes of data stored which could be used for various purposes around the business, as well as newly accessible hard-to-process sources such as web data, image data, and social media data.
This is where those who work in Data Engineering become extremely valuable, as they develop Data Platforms that allow this kind of volume to be processed and used by Data Scientists and analysts across the business.
WHY
HARNHAM?
Our esteemed position as a global leader in data recruitment stems from our deep expertise in Big Data and Data Engineering, supported by hundreds of dedicated specialists located across the United States, Europe, and the United Kingdom.
Our extensive network enables us to deliver exceptional recruitment services, expertly matching our clients with top-tier Data Engineering talent.
We acknowledge the distinct challenges and requirements of each organization's data infrastructure.
Therefore, our recruitment strategies are meticulously tailored, emphasizing a thorough understanding of and alignment with your specific business objectives and data technological needs.
OUR
SERVICES
-
Permanent and Contract Recruitment: We offer both permanent and contract recruitment solutions, adapting flexibly to support the dynamic requirements of your Big Data and Data Engineering projects.
Â
-
Executive Search: Our executive search service focuses on identifying and securing top-tier leaders in the Data Engineering domain who can advance your strategic goals and technological advancements.
Â
-
Industry-Specific Expertise: Our operations span multiple industries, providing specialized recruitment solutions tailored to the distinctive challenges and opportunities of your sector.
Â
Contact us today to discover how our custom talent solutions can bolster your organization's Data Engineering prowess.
JOBS
LATEST
OPPORTUNITIES
Harnham are a specialist Data & AI recruitment business with teams that only focus on niche areas.
Senior DataOps Engineer
Leeds
£62000 - £62000
+ Data Engineering
PermanentLeeds, West Yorkshire
To Apply for this Job Click Here
Senior Data Ops Engineer
Based: Leeds – hybrid
Salary: up to £62,000
I’m partnered with an established FS company who are scaling their cloud-native data platform and building a modern, centralised data function to better support a wide community of Data Scientists, Analysts, and federated Data Engineers across the organisation.
They are looking for a Senior DataOps Engineer to help shape how data pipelines are run, monitored, governed, and optimised at scale.
This is an opportunity to join a growing team working at the heart of the organisation’s data transformation, improving platform efficiency, enabling self-service, and ensuring data pipelines operate with the same discipline as production software.
The Role
As a Senior DataOps Engineer, you’ll take a strategic, high-level view of the data platform while still diving deep when needed. You will focus on observability, automation, pipeline performance, operational excellence, and cloud cost optimisation.
You’ll work cross-functionally with Data Engineering, DevOps, and FinOps teams, helping ensure that data services are reliable, scalable, secure and cost-effective, and that federated teams across the organisation can self-serve with confidence.
What You’ll Be Doing
- Taking an overview of how pipelines run across the platform, improving performance and throughput
- Enhancing observability and monitoring across Azure-based data workloads
- Identifying bottlenecks and opportunities to streamline operational processes
- Using scheduling/orchestration tools to optimise workflows and improve run times
- Treating data pipelines like production-grade software with robust monitoring, automation, and scalability in mind
- Supporting incident management and helping federated teams resolve issues efficiently
- Driving efficiency through automation and reduction of manual operational overhead
- Working with FinOps practices to optimise spend and evaluate cost-performance trade-offs
- Advocating for better platform usage, adoption, and operational best practices
Tech Stack & Skills
Essential
- Strong experience working with Azure cloud platform
- Background in data engineering and building/maintaining data pipelines
- Experience with pipeline monitoring, observability, and incident troubleshooting
- Strong automation mindset and ability to build resilient, self-healing data workflows
Nice to Have
- Knowledge of FinOps principles and cloud cost optimisation
- Experience with orchestration tools such as Azure Data Factory, Databricks Workflows, or Airflow
- Exposure to containerisation tools (Kubernetes, Docker)
- Experience with data cataloguing tools
- Familiarity with unit testing in data pipelines
- Awareness of MLOps practices
To Apply for this Job Click Here
Senior Data Engineer
Manchester
£80000 - £95000
+ Data Engineering
PermanentManchester, Greater Manchester
To Apply for this Job Click Here
Senior Data Engineer
Remote first (occasional travel to office)
Salary: up to £95,000
I’m partnered with a fast-scaling retailer who are looking to bring a Senior Data Engineer into their team. With over 40M+ global users and rapid year-on-year growth, this organisation is investing heavily in data to power smarter decisions, richer insights, and long-term product innovation.
You’ll work closely with internal teams to design, build, and optimise data models and ingestion pipelines, contributing to a new data lake and a robust gold-layer architecture. You’ll also have the chance to shape ML-ready pipelines and mentor more junior data engineers.
What you’ll be doing
- Building scalable data models and ingestion pipelines for a new data lake
- Working across multiple data domains and supporting internal partners
- Designing and delivering high-quality datasets for BI, insights, and analytics use cases
- Collaborating with product, engineering, and analytics stakeholders
- Supporting third-party data ingestion workflows
- Owning end-to-end delivery of production-grade data solutions
- Potential involvement in ML pipeline development
- Mentoring less experienced team members and fostering a culture of knowledge sharing
- Communicating clearly and effectively with diverse technical and non-technical audiences
Tech stack
- Python, Airflow, Terraform
- Azure (but open to other cloud experience)
- DBT (nice to have)
- Exposure to ML pipelines beneficial
To Apply for this Job Click Here
Senior Software Engineer – CDN & Go
San Francisco
$190000 - $240000
+ Data Engineering
PermanentSan Francisco, California
To Apply for this Job Click Here
Senior Software Engineer – Go (CDN and Distributed Systems)
Remote, US or Canada
$190-204k base + RSUs
A leading cloud infrastructure company is hiring a Senior Software Engineer with strong Go and distributed systems experience to help build and scale its next generation global CDN. This team operates at massive scale, handling billions of requests per day and powering some of the fastest, most reliable experiences on the web.
The Role
You will design and build high performance distributed systems that make the internet faster and more resilient. As part of the CDN Infrastructure group, you will work on projects such as:
- Evolving an internal CDN into a production ready customer platform
- Designing systems that support safe, incremental customer migration
- Scaling redirect handling from thousands to billions of requests
- Implementing HTTP/3 and exploring post quantum cryptography
- Driving performance improvements across the edge network
This role is ideal for engineers who enjoy deep technical challenges and want to influence how modern web infrastructure operates at global scale.
What We Are Looking For
- Strong experience with distributed systems, CDN, caching, or networking infrastructure
- Production Go experience. C and C plus plus backgrounds transfer well
- Solid understanding of the HTTP layer and cloud networking
- Experience operating systems at scale in a cloud or infra environment
- Bonus: Linux tuning, performance optimization, or open source work
To Apply for this Job Click Here
Consulting Data Engineer (Remote)
New York
$75 - $95
+ Data Engineering
ContractNew York
To Apply for this Job Click Here
Consulting Data Engineer – Professional Services
As a Consulting Data Engineer, you’ll be at the forefront of helping enterprises transform how they manage and leverage big data and AI. In this role, you’ll partner with clients to help design modern data architectures, guide migrations, and enable advanced analytics solutions that drive real business outcomes.
You won’t just deliver technology, you’ll deliver innovation. You’ll help organizations turn data into actionable insights. If you thrive in dynamic environments, love solving complex challenges, and want to make a measurable impact, this is the role for you.
What You’ll Do
- Lead high-impact projects that enable clients to modernize their data platforms and accelerate AI adoption.
- Design and implement scalable, cloud-based architectures that power advanced analytics and machine learning.
- Collaborate with technical teams and business stakeholders to ensure seamless delivery and measurable success.
- Act as a trusted advisor, guiding clients through migrations, deployments, and best practices for big data and AI.
- Share insights and feedback to continuously improve solutions and shape the future of data innovation.
What We’re Looking For
- 6+ years of experience in data engineering, analytics, or cloud architecture.
- Strong coding skills in Python or Scala.
- Expertise in at least one major cloud platform (AWS, Azure, or GCP) and familiarity with others.
- Deep knowledge of distributed computing frameworks (e.g., Apache Spark).
- Experience with CI/CD, MLOps, and designing end-to-end data solutions.
- Databricks experience is a *MUST*.
- Exceptional communication and problem-solving skills.
This is a Contract Role. The initial term will be six months, with a strong possibility for extension, and the workload will be 40 hours per week.
Why Join Us?
You’ll work on projects that shape the future of data and AI. You’ll collaborate with passionate experts, learn continuously, and make an impact that matters. If you’re ready to help organizations harness the power of data, apply here.
**WE WIL NOT BE UTILIZING THIRD PARTY AGENCIES FOR THIS ROLE**
To Apply for this Job Click Here
Senior Data Engineer
Accrington
£48000 - £55000
+ Data Engineering
PermanentAccrington, Lancashire
To Apply for this Job Click Here
Senior Data Engineer
Location: Accrington – hybrid
Salary: £48,000-£55,000
About the Role
My client is seeking a talented Senior Data Engineer to join a growing data function and play a key role in modernising data platforms, driving best practices, and mentoring junior colleagues. This role is ideal for someone who can hit the ground running, take ownership of key projects, and contribute to a collaborative and business-facing team.
You’ll work across a modern data stack centred on Snowflake, SQL, Python, and DBT, while also contributing to cloud-based ingestion pipelines and regulatory reporting initiatives.
What You’ll Be Doing
- Leading data engineering projects and acting as a senior point of contact within the team.
- Working closely with business SMEs and stakeholders to understand requirements and influence decisions.
- Designing and building scalable ingestion pipelines into Snowflake.
- Mentoring junior engineers and supporting team development.
- Working with tools such as Jira, Azure DevOps, AWS S3 (for file storage), and DBT (nice to have).
Tech Stack & Skills
Essential:
- Strong SQL and Python skills
- Solid experience with Snowflake
- 3+ years in Data Engineering, including at least 1 year in a senior or lead capacity
- Experience leading projects and communicating with senior stakeholders
- Strong soft skills and the ability to contribute to senior-level discussions
Nice to Have:
- Fundamental understanding of DBT
- Experience with AWS (especially S3), Azure DevOps, Jira
- Experience with data governance and regulatory reporting
To Apply for this Job Click Here
Senior Software Engineer – Lua
San Francisco
$200000 - $280000
+ Data Engineering
PermanentSan Francisco, California
To Apply for this Job Click Here
Senior Software Engineer – Lua
Remote (US) | $200,000 – $280,000 base + equity + benefits
A market leader in cloud infrastructure and developer tooling is seeking a Senior Software Engineer to help design, scale, and optimize a next-generation global CDN. This team builds the backbone of high-performance web delivery-handling billions of requests per day and powering some of the fastest, most reliable online experiences in the world.
The Role
As part of the CDN Infrastructure team, you’ll build and enhance distributed systems that make the internet faster and more resilient. You’ll design large-scale systems, write production-grade code, and work closely with product and infrastructure engineers to deliver high-impact improvements in routing, caching, and network performance.
What You’ll Do
- Design, implement, and optimize new CDN features and edge performance improvements
- Diagnose and reduce latency across a global network serving petabytes of data
- Improve reliability and scalability for billions of daily requests
- Build internal tools and workflows for infrastructure developers (primarily in Lua)
- Collaborate across teams to deliver next-generation web infrastructure
What We’re Looking For
- 5+ years of professional experience with Lua
- Deep understanding of HTTP, web servers, and CDN architecture
- Proven experience with distributed systems and large-scale performance optimization
- Strong problem-solving skills and a passion for experimentation
- Bonus: experience with Golang or other backend languages
Why Join
- Work on systems that serve petabytes of data and billions of requests daily
- Competitive compensation including equity and full health coverage
- Flexible, remote-friendly work environment
- Opportunity to shape the next generation of internet-scale infrastructure
To Apply for this Job Click Here
Staff Data Platform Engineer
San Francisco
$200000 - $245000
+ Data Engineering
PermanentSan Francisco, California
To Apply for this Job Click Here
Staff Data Platform Engineer
US Remote
$200-245k base + Equity
We’re working with a rapidly expanding consumer tech startup that recently raised $125M in Series D funding. As they continue to scale, they’re looking to hire a Staff Data Engineer to join their growing data team. If you have a strong foundation in software engineering, a collaborative mindset, and a passion for driving data strategy, this is the opportunity for you!
The Role:
As a Staff Data Engineer, you’ll be a crucial part of the team, ensuring data is accessible, reliable, and actionable across the company. Working alongside engineers, data scientists, and business teams, you’ll help build scalable data platforms that drive key business initiatives and enable data-driven decision-making.
What You’ll Do:
- Collaborate with engineering, data science, and business teams to understand data needs and implement effective solutions aligned with business objectives.
- Design and build scalable data infrastructure and pipelines to handle high volumes of data as the company grows.
- Lead the design and optimization of data architecture, ensuring it’s secure, high-performing, and scalable.
- Improve and streamline data pipelines to provide fast, reliable access to high-quality data.
- Contribute to the design of reporting frameworks and analytics tools that empower teams to make data-driven decisions.
- Work with security teams to ensure data privacy and compliance standards are met.
- Mentor junior engineers, share expertise, and promote best practices across the team.
- Stay updated on new technologies and suggest improvements to enhance processes and tool efficiency.
You’re a Great Fit If:
- You have 7+ years of experience in data engineering.
- You’ve worked at a leading consumer tech company known for engineering excellence.
- You’re an expert in SQL, Python, and data platform development.
- Nice-to-have: Familiarity with AWS, Kubernetes, Redshift, Athena, S3, and Airflow.
- You have hands-on experience with complex data infrastructures and optimizing data pipelines.
- You’ve led large-scale data projects that had a significant impact on your organization.
- You thrive in collaborative environments and have strong communication skills.
- You have an ownership mindset and take responsibility for delivering results.
To Apply for this Job Click Here
AI Software Engineer
Remote
$120000 - $160000
+ Data Engineering
PermanentOhio
To Apply for this Job Click Here
AI Software Engineer
Remote (US – ET/CT Time Zone)
$120K-$160K + Benefits
I’m partnering with a global leader in digital learning technology that’s transforming how millions of students and educators experience personalized, AI-powered learning. Following major investment in generative AI, the company is scaling its AI Platform Engineering team to build the next generation of adaptive learning tools.
This is a hands-on engineering role focused on building production-grade generative AI applications that power multiple products across the business.
The Role:
You’ll design and build scalable, full-stack systems that integrate large language models and AI capabilities into digital learning platforms. Working closely with data scientists, product managers, and engineers, you’ll help take GenAI prototypes from experimentation to production-driving forward real-world AI impact.
Key Responsibilities
- Develop and maintain AI-powered applications and platform services that are reliable, scalable, and secure
- Build and optimize LLM-based and RAG-powered solutions using frameworks such as LangChain or LangGraph
- Collaborate with Data Science teams to productionize models and streamline deployment pipelines
- Lead and contribute to technical design, architecture, and CI/CD improvements
- Ensure applications meet accessibility (WCAG 2.2 AA), performance, and security standards
- Stay current on the latest advancements in GenAI and advocate for best practices across the team
About You
- 5+ years of professional software engineering experience
- Strong full-stack skills across:
- Backend: Python, Node.js, or Go
- Frontend: Angular or React
- Databases: MySQL/PostgreSQL + NoSQL (e.g., DynamoDB)
- Cloud/Infra: AWS or Azure, Terraform, CI/CD tools
- Hands-on experience building AI/GenAI-enabled applications (Azure OpenAI, Amazon Bedrock, etc.)
- Practical understanding of RAG techniques, vector databases, and AI orchestration tools
- Strong system design, performance optimization, and collaboration skills
- Passionate about building impactful technology and staying ahead in the AI space
To Apply for this Job Click Here
Senior Software Engineer – Go
San Francisco
$190000 - $240000
+ Data Engineering
PermanentSan Francisco, California
To Apply for this Job Click Here
Senior Software Engineer – Go (CDN / Distributed Systems)
Remote – US or Canada
$190-240k base + RSUs
A market leader in cloud infrastructure and developer tooling is seeking a Senior Software Engineer (Go) to help design, scale, and optimize a next-generation global CDN.
This team builds the backbone of high-performance web delivery-handling billions of requests per day and powering some of the fastest, most reliable online experiences in the world.
If you’re passionate about distributed systems, high-performance networking, and building software that powers millions of developers, this is an opportunity to have a direct impact on how the web runs at scale.
The Role
As part of the CDN Infrastructure group, you’ll build and enhance distributed systems that make the internet faster and more resilient. You’ll design large-scale systems, write production-grade code, and collaborate closely with product and infrastructure teams to deliver high-impact improvements in routing, caching, and network performance.
You’ll work across key initiatives that include:
- Evolving an internal CDN framework into a fully production-grade product that customers can use directly.
- Designing systems that enable incremental customer migration onto the platform.
- Scaling redirect handling from thousands to billions of requests per site.
- Adding HTTP/3 support and exploring post-quantum cryptography for next-generation security.
- Driving performance optimization across the edge network and core infrastructure.
This is a role for engineers who love building systems that operate at global scale, thrive on technical depth, and want to shape the future of web infrastructure.
What We’re Looking For
We’re looking for backend engineers with strong distributed systems experience and the ability to design, implement, and scale complex infrastructure.
You’ll bring:
- Proven experience with distributed systems, CDN, caching, or ingress infrastructure.
- Go experience in production (C/C++ backgrounds highly transferable).
- Deep understanding of the HTTP layer and modern cloud networking.
- Experience running systems at scale in a cloud product or infrastructure environment.
- Bonus: experience in Linux kernel tuning, performance optimization, or relevant open-source contributions.
This role is ideal for engineers who are hands-on, performance-oriented, and eager to work on problems of scale.
Why Join
You’ll join a team that values technical excellence, autonomy, and impact-where your work directly shapes how developers build and deploy at the edge.
We operate globally, handle traffic at massive scale, and give engineers the ownership to build systems that redefine what’s possible in web performance and reliability.
We’re a remote-first organization that empowers engineers to make meaningful architectural decisions, ship quickly, and see their impact on real-world products at scale.
To Apply for this Job Click Here
CAN’T FIND THE RIGHT OPPORTUNITY?
GET IN TOUCH
today
If you can’t see what you’re looking for right now, send us your CV anyway – we’re always getting fresh new roles through the door.
Industry Hub
HARNHAM
News & Blog
With over 10 years experience working solely in the Data & AI sector our consultants are able to offer detailed insights into the industry.
Visit our Blogs & News portal or check out our recent posts below.
From Start-Up to Scale-Up: Hire Exceptional Data Talent in Two Weeks
By Guillian Eller, Manager AI/ML – Harnham You’re growing fast. You needed data talent yesterday. The reality…
Reskilling & Upskilling: What US Data Professionals Are Doing
By Luc Simpson-Kent, Managing Consultant – Harnham US Candidates want growth. Employers want retention. And in today’s…
What’s Driving Northern UK’s Data & AI Hiring in 2025?
by Jamie Smith, Senior Manager at Harnham, UK. At a glance The Northern data and tech market…
The Key to a Strong AI Team: Diversity
by Tom Brammer, Senior Manager – AI and Machine Learning US Team Every company wants to build…
Why US Companies Are Scaling AI with Contractors in 2025
Artificial intelligence has arrived in the workplace and it’s changing how companies are structured, resourced, and scaled. …
How Sacher.AI Strengthened Its GenAI Delivery in Just 12 Days
by Thomas Savidge, Principal Consultant – AI, Machine Learning and Data Science Sacher.AI, a UK-based innovation lab…
From Zero to AI: How One Investment Firm Built a World-Class Team in 18 Days
A global investment advisor managing over $268 billion across real estate, equity, and credit needed to build…
How to Maximise Your Data’s Value: The Use of AI in Advanced Analytics
By Roshni Baillie, Recruitment Consultant – Marketing & Insight, Harnham One thing I’m hearing again and again…
AI Governance Starts with People
By Kiran Ramasamy, Managing Consultant, Harnham When it comes to AI it is no longer a question…
15,000 Placements Later: What We’ve Learned About the Global Data & AI Talent Market
 By David Farmer, CEO, Harnham Global We haven’t just hit 15,000 placements, we’ve launched 15,000 careers,…
Testimonials
Client and
candidate
testimonials
A trusted partner of professionals
across the globe.
We understand the challenges our
customers face and offer the
recruitment solutions needed to drive
business success through Data &
AI.
Client – Director of People Operations
We’ve been collaborating with Harnham to recruit some key positions for our company, and they have been incredible partners. They team has been great and efficient. They understood our needs perfectly and provided us very talented candidates. We'll keep collaborating with them in the future. Thank you for your excellent job!
Client /
Director of People Operations
Client – Data Direction – Broadcast Media
I have worked with Harnham for over 2 years now as our main Analytics Recruiter partner and I am very pleased with the overall service and the people we have hired via Harnham. Harnham’s team is very good at understanding our recruitment needs and always come back with relevant and well scanned CVs, which saves me precious time.
Client /
Data Direction – Broadcast Media
Candidate – Market Research Analyst
Thanks to Harnham, I am now in a role that suits me. My recruiter was able to quickly identify the skills I had to offer and match me up with his client. It was a win-win for everyone. I was given great advice throughout the process, and I also appreciated the follow-ups that I received.
Candidate /
Market Research Analyst
Client – Director of Customer Data Science – Apparel
The team at Harnham have been outstanding in helping our company build a market-leading data science team. Harnham’s communication is great, and the recruiter’s take a consultative approach that’s really focused on provided value for both their clients and their candidates. They provide strong candidates that help us fill our roles with high achievers who are also a great fit for our culture.
Client /
Director of Customer Data Science – Apparel
Candidate – Data Analyst
I’ve had a great experience with Harnham. The recruiter provided great advice and ensured that I understood everything, and even followed up with me after I started my new role to see how I was settling in. I would recommend Harnham as a great recruitment company for anyone looking for a role in data and analytics.
Candidate /
Data Analyst
Candidate – Chief Data Officer
It was a real treat to work with Harnham. My recruiter consistently went above and beyond to ensure that I navigated through the recruitment process as smooth as possible. I had been in contact with other professional at Harnham in the past and I can attest that the level of professionalism is always top-notch.
Candidate /
Chief Data Officer
Candidate – Senior Data Engineer
I've been a Senior Data Engineer for about 4-5 years and Harnham is one of the best recruitment agencies I worked with. There was a time when my offer fell through and the recruiter I was working with at Harnham jumped in to the rescue and scheduled interviews for me within a couple of days. Really good agency and some great people.
Candidate /
Senior Data Engineer
Candidate – Digital Media and Analytics
I've recently got a short-term contract role through Harnham, and things couldn't have gone better! My recruiter was incredibly helpful throughout the entire process. They also provided incredible admin support so that I could focus on the actual job in hand, rather than having to spend too much time dealing with these matters.
