Senior Data Engineer
hirecrap
Job Description
- Design, develop, and maintain scalable ETL/ELT data pipelines for enterprise data platforms.
- Develop data engineering solutions using Microsoft Fabric, Azure Data Factory, Databricks, PySpark, Spark SQL, and Python.
- Build data pipelines for structured, semi-structured, and unstructured data.
- Work with Microsoft Fabric components such as Fabric Data Factory, Lakehouse, Warehouse, Dataflows, and Notebooks.
- Develop and optimize Databricks notebooks, Spark jobs, Delta Lake pipelines, and Databricks Workflows.
- Implement Medallion Architecture and modern Lakehouse/Data Platform architectures.
- Integrate data from multiple sources including SQL Server, Oracle, PostgreSQL, APIs, cloud storage, ERP systems, and SaaS applications.
- Design and implement efficient data models, data warehouses, data lakes, and Lakehouse solutions.
- Optimize data pipelines, Spark jobs, SQL queries, clusters, and storage for performance and cost efficiency.
- Implement data quality, validation, monitoring, governance, security, and compliance practices.
- Develop and maintain CI/CD pipelines using Azure DevOps/GitHub and Git.
- Troubleshoot production issues and provide root-cause analysis and performance improvements.
- Collaborate with solution architects, business analysts, data scientists, and BI teams.
- Participate in technical design, code reviews, documentation, and knowledge-sharing initiatives.
- Mentor junior engineers and contribute to technical best practices within the team.