Lead Data Engineer (Azure PaaS & Product Engineering) (10+ Years of Exp)

3pillarglobal

India Remote permanent

Posted: December 22, 2025

Required Skills

Job Description

We are building a customer-facing data platform from the ground up. The platform requires real-time data ingestion, high availability, and integration with Generative AI.
You will lead architecture decisions on Azure, sitting between Backend Engineering and Data Engineering. You will own a code-first approach: Python-based pipelines and APIs, not drag-and-drop ETL tools.

Responsibilities:
• Design event-driven ingestion architectures. Select appropriate Azure compute (Functions, Durable Functions, containers) based on workload characteristics.
• Design OLTP schemas for sub-second application retrieval, not just offline reporting.
• Architect data flows for RAG workflows. Ensure unstructured text is processed and indexed for vector search.
• Build reusable Python frameworks. Enforce unit testing, CI/CD, and code reviews within the data team.
• Own synchronization between operational databases (SQL/NoSQL) and analytical workloads. Design for consistency, replay capability, and clear separation between serving and analytical layers.

Required Skills:
• 10+ years in IT, with 5+ years as a Lead or Architect.
• Software engineering background (APIs, microservices) before moving into data. You understand that cloud data engineering is software development.
• Expert Python skills for backend processing, API integration, and automation. Clean, testable, production-grade code.
• Hands-on Azure experience: Functions (Consumption/Flex), Event Grid, Event Hubs. You understand tradeoffs between serverless, container-based, and dedicated compute.
• Experience building pipelines that feed LLMs or vector databases.
• Lakehouse architecture: medallion patterns, Delta Lake semantics, separation of storage and compute. Experience with Microsoft Fabric or Azure Databricks.
• Data modeling for both transactional systems and analytics. You know the difference between designing for an app vs. a data lake.

Good to Have:
• DP-203, DP-600, or Databricks Data Engineer Associate certification.
• Azure AI Search experience.
• Commercial SaaS product background.

Tech Stack:
• Python, SQL, PySpark, Spark, Azure Data Lake (ADLS Gen2), Azure Functions, Event Grid, Event Hubs, Delta Lake, Microsoft Fabric, Azure Databricks, REST APIs, Kafka, Airflow, Docker, Kubernetes, Git, Terraform, Data Pipelines, Streaming, Big Data, Data Governance, Data Quality

Regards
Rajan Paul(Linkedin Profile)
Talent Acquisition
3Pillar Global

Why Apply Through MisuJob?

AI-Powered Job Matching: MisuJob uses advanced artificial intelligence to analyze your skills, experience, and career goals. Our matching algorithm compares your profile against thousands of job requirements to find positions where you have the highest chance of success. This saves you hours of manual job searching and ensures you only see relevant opportunities.

One-Click Applications: Once you create your profile, applying to jobs is effortless. Your resume and cover letter are automatically tailored to highlight the most relevant experience for each position. You can apply to multiple jobs in minutes, not hours.

Career Intelligence: Beyond job matching, MisuJob provides valuable career insights. See how your skills compare to market demands, identify skill gaps to address, and understand salary benchmarks for your experience level. Make data-driven decisions about your career path.

Frequently Asked Questions

How do I apply for this position?

Click the "Register to Apply" button above to create a free MisuJob account. Once registered, you can apply with one click and track your application status in your dashboard.

Is MisuJob free for job seekers?

Yes, MisuJob is completely free for job seekers. Create your profile, get matched with jobs, and apply without any cost. We help you find your dream job without any hidden fees.

How does AI matching work?

Our AI analyzes your resume, skills, and experience to understand your professional profile. It then compares this against job requirements using natural language processing to calculate a match percentage. Higher matches mean better fit for the role.

Can I apply to jobs in other countries?

Absolutely. MisuJob features jobs from companies worldwide, including remote positions. Filter by location or look for remote opportunities to find jobs that match your preferences.

Ready to Apply?

Join thousands of job seekers using MisuJob's AI to find and apply to their dream jobs automatically.

Interested in this position?