Data Engineer

Design, develop, and maintain data ingestion pipelines using Azure Data Factory (ADF).

Develop pipelines to extract data from REST APIs, including handling:

·       Authentication mechanisms (OAuth, API Keys, Bearer Tokens)

·       Pagination

·       Incremental and historical data loads (multiple months of data)

·       Error handling and retries

Load and transform data into Azure target systems such as:

·       Azure Blob Storage

·       Azure SQL Database

Develop and optimize SQL queries, stored procedures, views, and data transformations.

Handle ad hoc data extraction, transformation, and reporting requests from business stakeholders.

Monitor scheduled pipelines, investigate failures, and perform production support activities.

Maintain source code, pipelines, and deployment artifacts using Azure DevOps with proper version control and branching strategies.

Collaborate with business users to understand reporting requirements and build interactive Power BI dashboards and reports.

Ensure data quality, consistency, and completeness across ingestion and reporting processes.

Participate in code reviews and follow development best practices.