Data Engineer
hcahr
Job Description
- Design, develop, maintain, and support ETL/ELT data pipelines for the HIM GCP Datamart. Extract, transform, and load data from Teradata, SQL Server, GCP Datamarts, and other enterprise data sources into HIM data platforms.
- Develop and support data workflows using Airflow, DAGs, Dataproc, Python, PowerShell, SSIS, SQL, and related data integration tools.
- Create, schedule, monitor, and troubleshoot Airflow DAGs for recurring data ingestion, transformation, validation, reconciliation, and delivery processes.
- Develop and optimize SQL queries, stored procedures, views, functions, and transformation logic across SQL Server, Teradata, PostgreSQL, BigQuery, and other relational platforms.
- Support on-premise to GCP data integration using Google Cloud Dataproc and related GCP services for large-scale data processing and transformation.
- Monitor data pipelines for failures, delays, performance issues, data quality problems, schema changes, and source system impacts.
- Troubleshoot production data issues, including missing data, late-arriving files, failed jobs, transformation errors, data mismatches, and performance bottlenecks.
- Perform root cause analysis and implement long-term fixes to improve pipeline reliability, reduce recurring failures, and improve operational stability.
- Implement data validation, reconciliation, and quality checks to ensure HIM data is complete, accurate, consistent, and timely.
- Collaborate with Parallon HIM Operations, source system owners, and technical teams to understand data needs, dependencies, refresh schedules, reporting impacts, and business priorities.
- Create and maintain data pipeline documentation, data flow diagrams, source-to-target mappings, job schedules, support procedures, and operational runbooks.
- Participate in release management, change control, defect resolution, enhancement delivery, incident response, and production support activities.
- Improve automation, monitoring, alerting, logging, and operational visibility across HIM data pipelines.
- Ensure data engineering solutions follow enterprise standards for security, privacy, compliance, data governance, change control, and production readiness.
- Identify opportunities to modernize legacy ETL processes and improve scalability, maintainability, reliability, and cost efficiency.