Senior Azure Data Engineer

Design, develop, and maintain scalable and reliable data pipelines using Azure Databricks and Azure Data Factory (ADF).

Develop data processing solutions using Python, PySpark, SQL, and Apache Spark.

Build and maintain Delta Lake tables and scalable data models for analytics and BI workloads.

Implement and manage Databricks Unity Catalog for data governance, security, access control, and data discovery.

Optimize Databricks jobs, Spark workloads, SQL queries, and data pipelines for performance, scalability, and cost efficiency.

Apply Apache Spark internals and tuning techniques to improve distributed data processing performance.

Develop complex and optimized SQL queries, including efficient joins and processing of large datasets.

Design and implement ETL/ELT workflows using Azure Data Factory.

Work with Azure data services including Azure Data Lake Storage (ADLS), Azure Synapse Analytics, and Azure SQL Database.

Implement data quality checks, validation, reconciliation, monitoring, and error-handling mechanisms.

Implement data governance, security, access control, and data lineage across data platforms.

Troubleshoot pipeline failures and performance issues and ensure high availability and reliability of data solutions.

Collaborate with business stakeholders, data architects, analysts, and engineering teams to understand requirements and deliver effective data solutions.

Automate manual processes and continuously improve data engineering workflows and operational efficiency.

Mentor junior data engineers and promote best practices across Azure and Databricks data engineering.