Senior Data Engineer
scalis
Job Description
- Design, develop, and maintain scalable, secure, and high-performing data pipelines using Microsoft Fabric, Azure Data Factory, PySpark, and Databricks.
- Create and manage data pipelines to ingest, process, and transform data from various sources into the lakehouse.
- Collaborate with data architects, business analysts, and other stakeholders to understand data requirements and translate them into scalable solutions.
- Build and optimize data models and data warehousing solutions to support business intelligence and advanced analytics, implementing techniques such dimensional modeling, normalization, creating surrogate keys, utilizing Slowly Changing Dimensions, partitioning, and aggregation.
- Perform data transformation and pipeline orchestration using Fabric Notebooks to cleanse and prepare data for report consumption.
- Implement and maintain security measures to protect data assets
- Ensure data quality, integrity, and governance across all stages of the data lifecycle.
- Optimize performance of data ingestion, transformation, and loading processes.
- Proactively identify areas for process improvement, automation, and cost efficiency within the data pipeline infrastructure.
- Provide technical mentorship to junior engineers and collaborate on the adoption of best practices in data engineering.
- Maintain comprehensive documentation of data systems, processes, and workflows.