LEAD DATA ENGINEER

happiestminds

Bengaluru 8 Years Exp Posted 19d ago

Technical Leadership and Design Support

Work closely with solution and data architects to understand high-level architecture and translate it into detailed technical designs.
Support the architect in evaluating solution options, integration patterns, data models, and pipeline-design approaches.
Prepare and review low-level design documents, data-flow diagrams, source-to-target mappings, interface specifications, and technical implementation plans.
Make appropriate technical decisions related to pipeline architecture, data processing, performance, scalability, security, and maintainability.
Identify technical dependencies, risks, assumptions, and constraints during design and development.
Conduct technical discussions with business analysts, source-system teams, QA teams, DevOps teams, and other engineering stakeholders.
Estimate development efforts and support sprint planning, task allocation, and delivery tracking.
Ensure that technical solutions comply with organizational architecture, security, governance, and data-engineering standards.

Hands-on Development

Design, develop, test, and maintain scalable ETL/ELT data pipelines using Databricks and PySpark.
Develop complex data-transformation logic using PySpark, Spark SQL, and SQL.
Build and maintain Delta Lake tables following Bronze, Silver, and Gold layers of the Medallion Architecture.
Implement Delta Lake capabilities, including:
- MERGE, upsert, and deduplication operations
- Schema enforcement and schema evolution
- Time Travel and version management
- OPTIMIZE and VACUUM
- Incremental and change-based data processing
Develop reusable PySpark and Python utilities, frameworks, common components, and coding templates.
Develop and manage Databricks Workflows and Jobs for orchestration, scheduling, dependency management, retries, and alerting.
Write complex SQL queries for data transformation, reconciliation, validation, analysis, and troubleshooting.
Integrate Databricks pipelines with external systems using REST APIs.
Design and support inbound and outbound file integrations using SFTP.
Process structured and semi-structured data in CSV, JSON, Parquet, and Delta formats.
Work with AWS services used within the data engineering ecosystem, particularly Amazon S3.
Implement appropriate exception handling, logging, auditing, monitoring, alerting, and restart/recovery mechanisms.

Code Review and Engineering Quality

Conduct detailed code reviews for PySpark, Python, Spark SQL, and SQL components developed by the team.
Ensure code is functionally correct, reusable, maintainable, secure, and optimized for performance.
Establish and enforce coding standards, naming conventions, repository structures, branching practices, and documentation guidelines.
Review pipeline designs and code for partitioning, joins, caching, shuffle operations, data skew, file sizing, and query execution.
Ensure adequate unit testing, data validation, reconciliation, error handling, and audit controls.
Identify technical debt and drive code refactoring and continuous engineering improvements.
Ensure AI-generated code is properly reviewed, tested, secured, and optimized before it is committed or deployed.
Participate in pull-request reviews and ensure that identified review comments are addressed before code approval.

Team Mentoring and Delivery Support

Provide technical guidance and day-to-day support to junior and mid-level data engineers.
Allocate technical tasks based on developer capability, delivery priority, and complexity.
Mentor junior developers in Databricks, PySpark, Spark SQL, Delta Lake, Python, and data-engineering best practices.
Conduct technical walkthroughs, knowledge-sharing sessions, pair-programming exercises, and code-quality reviews.
Help team members troubleshoot complex development, performance, and production issues.
Review developer estimates and monitor technical progress against sprint and release commitments.
Identify skill gaps and create structured learning and development plans for junior engineers.
Promote reusable solutions and consistent engineering practices across the development team.
Act as the primary technical point of contact for the offshore or delivery team.

Performance Optimization and Production Support