Azure Data Engineer
persistent
Job Description
- Design, develop, test, and maintain scalable data engineering solutions on Azure Cloud.
- Build and optimize batch and real-time data processing pipelines using Databricks and PySpark.
- Develop robust data ingestion, transformation, and integration frameworks for structured and unstructured data.
- Translate business and technical requirements into scalable, maintainable, and secure data solutions.
- Develop and support data processing workflows using Python and cloud-native services.
- Implement streaming data solutions using Kafka and modern event-driven architectures.
- Perform code reviews, unit testing, integration testing, debugging, and production support activities.
- Optimize data pipelines and processing frameworks for performance, reliability, and scalability.
- Collaborate with architects, product owners, business analysts, QA teams, DevOps teams, and data governance stakeholders.
- Ensure data quality, consistency, governance, and compliance throughout the data lifecycle.
- Support CI/CD implementation and automation for data engineering workloads.
- Create and maintain technical documentation, reusable assets, and operational procedures.
- Contribute to engineering best practices, innovation initiatives, and continuous improvement efforts.