Data Engineer
aptean
Job Description
- Design and build robust, scalable data ingestion pipelines using Microsoft Fabric (Pipelines, Dataflows, Notebooks) to integrate data from Business Applications, databases, files, and APIs into the Lakehouse.
- Perform deep source system analysis to define ingestion strategies that ensure data reliability, consistency, and observability, while applying metadata-driven design for automation.
- Develop and maintain Delta Tables using the medallion architecture (bronze/silver/gold) to systematically cleanse, enrich, and standardize data for downstream consumption.
- Implement comprehensive data quality checks (nulls, duplicates, schema drift, outliers, SCD types) and ensure data integrity across all transformation layers in the Lakehouse.
- Apply governance practices including schema versioning, data lineage tracking, role-based access control (RBAC), and audit trails to ensure compliance, traceability, and secure data access.
- Structure the gold layer and semantic model to support AI/ML use cases, ensuring datasets are enriched, contextualized, and optimized for AI agent consumption.
- Develop and maintain AI-ready run flows and access patterns to enable seamless integration between the Lakehouse and AI agents for tasks such as prediction, summarization, and decision automation.
- Leverage DevOps best practices for pipeline versioning, testing, deployment, and monitoring; proactively detect and resolve data integration and processing issues.