Senior Data Engineer

This role is for one of the Weekday's clients

Min Experience: 12 years

Location: Bengaluru

JobType: full-time

Requirements:
Role Overview

• Oversee the creation of cutting-edge cloud-based data platforms and data applications.
• Provide scalable data engineering solutions that facilitate investment analytics and business insights.
• Architect and deploy Lakehouse architecture utilizing Databricks.
• Develop data pipelines adhering to the Medallion Architecture (Bronze, Silver, Gold layers).
• Lead and mentor a team of data engineers while promoting engineering best practices.

Key Responsibilities

• Design and construct scalable data pipelines and data platforms.
• Implement Lakehouse architecture with Databricks and cloud technologies.
• Create ETL/ELT pipelines and frameworks for data integration.
• Develop event-driven and real-time data pipelines.
• Guarantee data governance, security, and access controls.
• Establish CI/CD pipelines and automate infrastructure.
• Work closely with architects, product owners, and business stakeholders.
• Champion engineering best practices, including code reviews, testing, and version control.

Required Skills

• Extensive experience with Databricks and Lakehouse architecture.
• Familiarity with implementing Medallion Architecture (Bronze, Silver, Gold).
• Proficient in cloud data platforms such as Snowflake.
• Deep understanding of AWS services (Lambda, Glue, S3).
• Knowledge of streaming architecture utilizing Kafka.
• Solid programming abilities in Python and SQL.
• Experience with CI/CD pipelines and Infrastructure as Code (Terraform).
• Familiarity with workflow orchestration tools like Airflow or Control-M.
• Strong knowledge of RDBMS (Oracle, PostgreSQL, MSSQL).
• Exposure to NoSQL databases.
• Experience with GitHub, version control, and engineering best practices.

Skills

observability

lake house architecture

medallion architecture

Data bricks

pyspark

python