Senior Data Engineering Lead
unitedhealthgroup
Job Description
- Build reliable batch, micro-batch, and event-driven pipelines on AWS and Databricks
- Develop reusable ingestion frameworks for REST APIs, FHIR Bulk Export, HL7 interfaces, databases, SFTP/file exchange, JSON/NDJSON, CSV, XML, PDFs, and clinical text
- Implement scalable Spark/PySpark and SQL transformations, including schema inference/evolution, checkpointing, idempotency, retries, backfills, and replay
- Design and maintain Delta Lake and Apache Iceberg tables, including physical design, partitioning/clustering, compaction, file sizing, incremental reads/writes, performance tuning, and cost optimization
- Build curated healthcare data models and transformations using FHIR, HL7, OMOP CDM, and clinical terminology mappings
- Implement data-quality frameworks: schema validation, referential-integrity checks, business-rule testing, anomaly detection, reconciliation, completeness checks, and data-quality observability
- Build metadata and lineage capture across source systems, pipeline runs, code versions, transformation rules, mappings, and published data products
- Implement privacy-aware data processing for PHI, including access controls, masking, tokenization/pseudonymization, de-identification, and auditable handling patterns
- Deliver CI/CD pipelines, automated unit/integration/data tests, Terraform or CloudFormation, containerized services, monitoring, alerting, runbooks, and incident-recovery procedures
-
Integrate and optimize governed data consumption through Databricks SQL, Athena, Snowflake, Trino, or equivalent engines
- Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements