Lead Data Engineer

Technical Leadership & Development

 

•        Architect, build, and optimize scalable data pipelines using Databricks, PySpark, and Python

 

•        Write and review complex SQL queries for data transformation, validation, and performance tuning

 

•        Design and maintain data models and ETL/ELT workflows across Databricks environments

 

•        Support or lead development of ML models/pipelines for data products (feature engineering, model training, deployment, and monitoring)

 

•        Build data product applications using AI-assisted "vibe coding" tools (Claude Code, Cursor, or similar) to rapidly prototype, develop, and ship internal tools, dashboards, and data-driven applications

 

•        Establish and enforce best practices for code quality, version control, and CI/CD using Git

 

•        Champion the adoption of AI-assisted development tools across the team to improve developer velocity, code review efficiency, and documentation

 

•        Conduct code reviews, architecture reviews, and technical design sessions

 

Infrastructure & Platform Ownership

 

•        Own and maintain the infrastructure underlying data products — compute clusters, transactional databases, job orchestration, environments, deployment pipelines, and monitoring

 

•        Ensure data pipeline reliability, observability, and cost optimization across Databricks

 

•        Manage access controls, environment configuration, and platform scaling for data product apps

 

•        Troubleshoot infrastructure issues end-to-end (compute, storage, networking, connectivity)

 

Team & People Management

 

•        Lead, mentor, and grow a team of 4 engineers, providing technical guidance and career development support

 

•        Conduct 1:1s, performance reviews, and skill-development planning for team members

 

•        Foster a culture of continuous learning and rapid upskilling in response to evolving tech/business needs

 

•        Encourage and enable the team to adopt new tools and frameworks quickly

 

Delivery & Process Management

 

•        Own sprint planning, backlog grooming, and sprint retrospectives for the team (Agile/Scrum)

 

•        Break down business requirements into well-scoped technical tasks and user stories

 

•        Track sprint velocity, manage blockers, and ensure timely delivery of data engineering initiatives

 

•        Coordinate with cross-functional stakeholders (analytics, product, business teams) to align priorities