Lead Azure Databricks Engineer

Who is HelloKindred?

HelloKindred are specialists in staffing marketing, creative and technology roles, offering a range of talent solutions that can be delivered on-site, remotely or hybrid.

Our vision is to make work accessible and people’s lives better. We do this by disrupting traditional employment barriers – connecting ambitious talent to flexible opportunities with trusted brands.

Anticipated Contract End Date/Length: December 31, 2026

Work set up: Hybrid

Our client in the Information Technology and Services industry is looking for a Lead Azure Databricks Engineer to drive the design, build, optimisation, and governance of enterprise data platforms within a highly regulated insurance environment. This is a 100% hands-on role requiring deep technical expertise combined with strong leadership capability.

The successful candidate will operate within complex, fast-paced settings and bring proven delivery experience aligned to Lloyd’s of London regulatory and operational requirements. This role will play a pivotal part in shaping cloud data strategy, enabling advanced analytics, and delivering secure, scalable, and high-performing solutions across the organisation.

What you will do:

• Lead the development and optimisation of large-scale data platforms using Azure services including ADF, Data Lake, Key Vault, and Azure Functions, alongside Databricks technologies such as Delta Lake, PySpark, and Unity Catalog
• Collaborate with underwriting, actuarial, delegated authority, bordereaux, exposure management, reinsurance, finance Solvency II, and risk teams to deliver compliant and auditable data solutions
• Translate business and regulatory requirements into scalable, secure cloud data architectures
• Own and resolve complex technical challenges with clarity, speed, and accountability
• Champion best-practice design patterns, Medallion architecture, Lakehouse principles, and data lifecycle management
• Implement CI/CD pipelines using Azure DevOps, including branching and merging strategies for data solutions
• Drive automation and cloud-native engineering principles across the platform
• Provide technical leadership and mentorship, setting engineering standards and uplifting team capability
• Identify opportunities to improve performance, reduce cost, and enhance resilience through continuous improvement initiatives
• Oversee end-to-end data engineering delivery, ensuring reliability, observability, data quality, and performance optimisation

• Expert-level engineering experience across Azure Data Services and Databricks in enterprise-scale environments
• Deep understanding of Delta Lake, Medallion architecture, distributed compute, Lakehouse patterns, data modelling, and optimisation
• Strong proficiency in Python, PySpark, and Spark SQL
• Experience implementing CI/CD for data engineering solutions using Azure DevOps
• Solid understanding of data governance, lineage, access controls, FinOps, and secure cloud engineering practices
• Proven experience delivering within the Lloyd’s of London market or similarly regulated insurance environments
• Excellent communication skills with the ability to engage senior stakeholders and lead cross-functional discussions
• Demonstrated ownership, accountability, and consistent delivery in high-expectation, fast-paced environments

All your information will be kept confidential according to EEO guidelines.

Candidates must be legally authorized to live and work in the country where the position is based, without requiring employer sponsorship.

HelloKindred is committed to fair, transparent, and inclusive hiring practices. We assess candidates based on skills, experience, and role-related requirements.

We appreciate your interest in this opportunity. While we review every application carefully, only candidates selected for an interview will be contacted.

HelloKindred is an equal opportunity employer. We welcome applicants of all backgrounds and do not discriminate on the basis of race, colour, religion, sex, gender identity or expression, sexual orientation, age, national origin, disability, veteran status, or any other protected characteristic under applicable law.