Data Engineering Manager

About Upshop:

Upshop is the foremost provider of a SaaS platform designed to streamline forecasting, ordering, production, and inventory optimization processes for food retailers. Its unified platform simplifies and enhances associate tasks, promoting smarter and more interconnected operations across Fresh, Center, DSD, and eCommerce departments. With over 450+ retailers and 50,000+ stores relying on its mission-critical operations platform globally, customers have witnessed substantial enhancements in sales, shrinkage reduction, food safety, and sustainability throughout their stores. 

At Upshop, we believe that great businesses are built by great people. Our People function is at the heart of our company’s growth, ensuring we attract, develop, and retain A Players who drive our mission forward.

Our Values:

• Extremely Accountable

• Customer Obsessed

• Always Innovating

• Demand Excellence

• Biased for Action

Data Engineering Manager

Role Overview

We are seeking an experienced Data Engineering Manager to lead the technical evolution of our Databricks Lakehouse. You will be responsible for the architecture, scaling, and reliability of our data products, ensuring that the platform provides high-quality, actionable data for retail and supply chain analytics. This is a leadership role that requires a balance of advanced systems design and hands-on operational rigor.

Key Responsibilities

🔹 Architecture & Platform Ownership

• Lakehouse Strategy: Own the end-to-end design and optimization of the Databricks Lakehouse architecture, leveraging Unity Catalog and Delta Lake for multi-tenant isolation.

• Scalable Ingestion: Drive multi-tenant ingestion strategies from Azure Table Storage, Cosmos DB, and APIs, focusing on watermark logic and idempotent pipelines.

• Framework Development: Lead the evolution of in-house PySpark-based ETL frameworks to support complex transformations and downstream consumption.

🔹 Operational Excellence & Data Observability

• Data Quality Validation: Implement comprehensive data validation frameworks and automated checks to ensure accuracy, completeness, and consistency across the Lakehouse.

• Observability & Monitoring: Drive the strategy for end-to-end data observability, using monitoring and alerting to proactively identify pipeline drifts or failures before they impact stakeholders.

• FinOps & Performance: Improve cost transparency through DBU tracking and compute profiling (Serverless vs. Classic); drive performance tuning via clustering and partitioning strategies.

🔹 DevOps & Technical Governance

• Deployment Integrity: Oversee CI/CD pipelines and Databricks Asset Bundles (DABs) to maintain parity between UAT and Production environments.

• Governance Standards: Implement version control, rollback strategies, and deployment governance to ensure a stable and auditable production environment.

🔹 Cross-Functional Leadership

• Strategic Partner: Partner with BI, Data Science, and Product teams to bridge business requirements into scalable, production-grade data solutions.

• Team Growth: Mentor and lead a distributed team of engineers, focusing on high-velocity delivery and technical excellence.

• Tenant Logic: Support customer onboarding by designing tenant-specific transformation logic that scales without increasing architectural complexity.

Qualifications

• 8+ years in Data Engineering; 3+ years in a leadership/management role.

• Expert in Workflows, Unity Catalog, and Delta Lake.

• Deep proficiency in PySpark, SQL, and Azure Data ecosystem (Storage, Cosmos DB).

• Proven experience managing multi-tenant or enterprise-scale platforms.

• Strong background in CI/CD, Infrastructure-as-Code, and Data Modeling.

Preferred Skills

• Experience with retail or supply chain data forecasting.

• Expertise in optimizing serverless compute environments.

• Experience maintaining production SLAs for customer-facing data dependencies.