Principal Data Engineer

arrow

San Francisco / $200000 - $240000 annum

INFO

Salary

SALARY:

$200000 - $240000

Location

LOCATION

San Francisco

Job Type
JOB TYPE

Permanent

Principal Data Engineer (L5) / Staff Data Engineer

Remote - United States
Base Salary: $220,000 - $240,000
Eligibility: U.S. Citizen or Green Card holder required


Overview

A leading digital product and AI engineering firm is seeking a Principal Data Engineer (L5) to join a high-impact, multi-disciplinary team building modern, enterprise-grade data platforms.

This is a senior technical leadership role suited for an experienced engineer who can define data strategy, make end-to-end platform decisions, and influence engineering direction across complex engagements. You will operate at the intersection of data engineering and AI, helping deliver production-ready systems at speed while maintaining high standards in performance, reliability, and security.


Why This Role Matters

You will play a critical role in shaping how AI-powered data systems integrate into real-world enterprise workflows. The team operates at high velocity, delivering production-grade solutions in accelerated timelines, requiring strong ownership, technical craftsmanship, and the ability to execute with precision.

This role provides visibility across platform architecture, client impact, and internal engineering standards-ideal for someone who wants to influence both technical direction and team capability.


Key Responsibilities

Data Architecture & Platform Ownership

  • Define and lead data architecture strategy across data lakes, warehouses, and pipelines
  • Design scalable, high-performance systems for batch and real-time data processing
  • Drive modernization efforts, optimizing for cost, performance, and scalability
  • Establish data contracts and event-driven architectures across distributed systems
  • Integrate data platforms with APIs, middleware, and third-party systems

AI & Advanced Data Systems

  • Design and implement data pipelines supporting production AI/ML systems, including:
    • Embeddings and vector databases
    • Retrieval-Augmented Generation (RAG) data preparation
    • Feature stores and model training/inference pipelines
  • Apply an AI-forward approach using modern development tools (e.g., Claude, Cursor)

Governance, Quality & Reliability

  • Define and enforce data governance, quality standards, and compliance frameworks
  • Implement monitoring, logging, and alerting systems across data pipelines
  • Contribute to CI/CD pipelines for data workflows and automated deployment

Leadership & Collaboration

  • Partner with leadership to shape data strategy and translate technical complexity into actionable decisions
  • Mentor and guide junior and mid-level engineers, elevating team capability
  • Establish engineering standards and best practices across data platforms
  • Collaborate with engineering, product, analytics, and AI teams to align on execution

Required Qualifications

  • 7+ years of experience in data engineering, including leadership of complex data platform initiatives
  • 7+ years of hands-on experience with AWS, with recent, production-level usage required
  • Strong background in distributed systems and large-scale data architecture
  • Expertise in Python, Scala, and SQL
  • Deep experience with cloud-native data platforms and enterprise data warehousing
  • Strong knowledge of data pipeline orchestration and transformation frameworks
  • Experience with streaming and real-time systems (Kafka, Kinesis, Pub/Sub, etc.)
  • Proven expertise in data modeling and scalable data design
  • Hands-on experience with governance, compliance, and data quality frameworks
  • Experience with containerization and CI/CD for data systems
  • Demonstrated experience building pipelines for AI/ML production systems
  • Strong communication skills with the ability to influence cross-functional stakeholders
  • Track record of mentoring engineers and leading technical decisions

Preferred / Nice-to-Have

  • Experience with data mesh, data fabric, or lakehouse architectures
  • Background in healthcare data systems
  • AWS certifications (e.g., AWS Certified Data Engineer - Associate)

Interview Process

  • 30-minute Culture Interview
  • 30-45 minute Technical Interview
  • Final Client Interview

CONTACT

Stephanie Landa

Principal Recruitment Consultant

SIMILAR
JOB RESULTS

4k-Harnham_DA copy
CAN’T FIND THE RIGHT OPPORTUNITY?

STILL
LOOKING?

If you can’t see what you’re looking for right now, send us your CV anyway – we’re always getting fresh new roles through the door.