Job Description

Lead Backend Software Engineer – Data Services

Federated Data Services | Distributed Systems | Apache Spark | DataFusion | Python | Kubernetes

Location: London (2 days per week onsite)

Rate: £700–£800 per day (Outside IR35)

Duration: Initial 6-month contract with likely extension

Overview

We're looking for an experienced Lead Backend Software Engineer to join a high-profile programme building secure, cloud-native data services that will support data-driven research and innovation at national scale.

This is a founding engineering role, offering the opportunity to shape the architecture, engineering standards, and core data services from the ground up. Working closely with the Engineering Manager and a small team of senior engineers, you'll establish the technical foundations for a complex federated data platform operating within a highly regulated environment.

We're looking for someone who enjoys building from first principles, thrives in ambiguity, and can balance strategic architectural thinking with hands-on software engineering. This role is centred around designing and building distributed data services rather than traditional application development.

Role & Responsibilities

Lead the design and development of scalable, cloud-native data services supporting a federated platform.

Define architectural patterns, engineering standards, and best practices that will scale across the wider organisation.

Design and build distributed server-side services using modern software engineering practices.

Architect and develop federated data services capable of securely querying and processing data across multiple environments.

Design high-performance data processing and streaming workflows using technologies such as Apache Spark, Apache DataFusion, or similar distributed query frameworks.

Design and implement Kubernetes-based services and containerised workloads.

Build secure APIs and backend services supporting authentication, authorisation, governance, and audit requirements.

Contribute to the design of distributed systems, data pipelines, and platform architecture.

Establish CI/CD pipelines, Infrastructure as Code, automated testing, and deployment practices.

Evaluate and integrate open-source technologies and distributed data frameworks.

Mentor engineers and promote engineering excellence across the team.

Work closely with architecture, product, security, and external stakeholders.

Skills & Experience

Essential

Experience as a Lead Engineer, Principal Engineer, Staff Engineer, Software Architect, or similar senior technical role.

Strong server-side software engineering experience, with flexibility across programming languages such as Python, Java, Go, Scala, C#, or similar.

Experience designing and building distributed data services or large-scale backend platforms.

Experience with Apache Spark, Apache DataFusion, Trino, Presto, or similar distributed data processing technologies.

Strong architecture and design experience across distributed systems and data services.

Experience designing scalable data processing or data streaming workflows.

Experience building cloud-native applications on AWS, Azure, or GCP.

Experience with Kubernetes, Docker, and containerised environments.

Experience building and operating CI/CD pipelines and Infrastructure as Code.

Strong understanding of distributed systems, scalability, resilience, and secure-by-design principles.

Comfortable making architectural decisions within evolving, fast-paced environments.

Excellent communication skills with the ability to work across technical and non-technical teams.

Desirable

Experience building federated data services or federated data platforms.

Experience within healthcare, life sciences, research, government, or other highly regulated environments.

Knowledge of Trusted Research Environments (TREs), Secure Data Environments (SDEs), or similar secure data platforms.

Experience with data governance, metadata services, data catalogues, or secure data access models.

Exposure to event-driven architectures and streaming technologies such as Kafka, Flink, or Spark Streaming.

Experience evaluating or contributing to open-source technologies