Data Engineer

Data Engineering

• Design and build production-grade data pipelines using Azure Data Factory and Azure Databricks.
• Develop scalable data architectures leveraging Azure SQL, Data Lake, and modern data modelling techniques (star/snowflake schemas).
• Implement data quality validation, reconciliation processes, and resolve operational issues.

Analytics & AI Enablement

• Use Python for data processing, transformation, and analytics.
• Develop and maintain Jupyter Notebooks for exploration and experimentation.
• Utilize MLflow for experiment tracking and model lifecycle management.

BI Enablement

• Deliver curated, analytics-ready datasets for Power BI and other reporting platforms.
• Optimize data models and structures for efficient reporting, dashboarding, and analytics consumption.

Engineering Practices

• Implement CI/CD pipelines for data engineering workflows.
• Ensure secure data handling practices, including sensitive data protection.
• Monitor, troubleshoot, and resolve pipeline failures and performance issues.

Required Skills & Qualifications

Strong expertise in:

• • Azure Data Factory
  • Azure Databricks
  • Azure SQL and Data Lake
  • Solid understanding of data modelling (star and snowflake schemas).
  • Experience building scalable and reliable data pipelines.
• Core Competencies
  • Strong problem-solving and analytical skills.
  • Ability to manage production-grade data environments.
  • Focus on data quality, scalability, and performance optimization.
• Nice to Have
  • Experience working in regulated data environments.
  • Exposure to Azure OpenAI or AI/ML use cases.
    • Knowledge of prompt engineering, evaluation frameworks, and experimentation techniques.