Senior Databricks Data Engineer

3-Month Contract | Outside IR35 | Fully Remote | £500 per day

We're supporting a business that is investing heavily in its data capabilities and is looking for an experienced Senior Databricks Data Engineer to take ownership of a growing cloud-based data platform.

This assignment will focus on building a robust and scalable data ecosystem that supports reporting, analytics and strategic decision-making across a range of business functions. You'll work closely with both technical and non-technical stakeholders to transform fragmented data sources into trusted, well-governed datasets that drive real business value.

This is a hands-on engineering role suited to someone who is comfortable leading technical decisions, establishing best practices and delivering production-grade Databricks solutions.

• Building and enhancing data pipelines using Databricks, PySpark and SQL.
• Designing and implementing scalable Lakehouse architectures using Delta Lake.
• Integrating data from a variety of platforms through APIs, file-based ingestion and third-party systems.
• Developing and maintaining Bronze, Silver and Gold data layers to support analytics and reporting requirements.
• Defining data governance standards, lineage, security controls and access management.
• Delivering robust ETL/ELT solutions across cloud environments.
• Establishing monitoring, testing and data quality frameworks to improve reliability and trust in data.
• Implementing CI/CD practices and automated deployment processes.
• Optimising Databricks workloads for performance, scalability and cost efficiency.
• Working closely with analytics, reporting and business teams to turn requirements into technical solutions.
• Supporting and mentoring other members of the wider data function.

• Strong commercial experience delivering enterprise-scale data engineering solutions on Databricks.
• Advanced Python, PySpark and SQL skills.
• Proven experience designing Lakehouse architectures and working with Delta Lake.
• Strong understanding of Databricks Workflows and production pipeline orchestration.
• Experience implementing data governance controls using Unity Catalog or similar tooling.
• Experience building cloud-native data solutions within Azure, AWS or GCP environments.
• Strong understanding of data modelling and analytical data structures.
• Experience with source control, automated testing and deployment pipelines.
• Knowledge of security, governance and compliance requirements when handling sensitive data.
• Ability to communicate effectively with business stakeholders and translate requirements into technical outcomes.

• Experience with Databricks Asset Bundles.
• Terraform and Infrastructure-as-Code experience.
• Azure DevOps and/or GitHub Actions.
• Power BI integration and optimisation.
• Experience working with HR, workforce, payroll or other sensitive operational datasets.
• Previous experience acting as a technical lead or subject matter expert within a Databricks environment.

You'll be a hands-on Data Engineer who enjoys taking ownership of platform design and engineering standards. You're equally comfortable discussing architectural decisions with technical teams as you are explaining data solutions to business stakeholders.

This contract would suit someone with deep Databricks expertise who wants to make an immediate impact within a business undergoing significant data transformation.

Contract Length: 3 months Initial Contract Location: Fully Remote (UK) Rate: Up to £500 per day Engagement: Outside IR35 Start: ASAP