Lead Data Engineer

- Lead the architecture and design of scalable, secure, and high-performance data engineering solutions for complex business requirements.

- Own end-to-end data platform initiatives, from technology evaluation and solution design to implementation, deployment, and production support.

- Define data ingestion, transformation, processing, storage, and consumption strategies across enterprise data ecosystems.

- Design and optimize large-scale batch and streaming data pipelines integrating data from multiple internal and external sources.

- Establish reusable data engineering frameworks, standards, design patterns, and best practices across projects.

- Lead the development of cloud-based data platforms leveraging AWS services such as S3, Redshift, EMR, Athena, Kinesis, and Lambda.

- Drive the design and implementation of modern data architectures using technologies such as Databricks and Snowflake.

- Provide technical leadership for ETL/ELT development, data warehousing, data lake, and data lakehouse initiatives.

- Design robust mechanisms for handling schema evolution, schema drift, data dependencies, and source-system changes.

- Establish data quality, validation, reconciliation, monitoring, and observability frameworks to ensure reliability of critical data pipelines.

- Define strategies for pipeline performance optimization, scalability, availability, and cost efficiency.

- Lead integration of data from supplier systems, landing zones, APIs, databases, files, and other heterogeneous sources.

- Evaluate and recommend appropriate technologies, tools, and frameworks based on scalability, performance, maintainability, and business requirements.

- Design workflow orchestration and dependency management using tools such as Airflow, Azkaban, or Luigi.

- Guide implementation of ETL solutions using platforms such as Talend, Informatica, or equivalent technologies.

- Work with NoSQL and relational databases to design appropriate data storage and processing solutions.

- Establish engineering practices around version control, CI/CD, testing, deployment, monitoring, and production support.

- Lead root-cause analysis for complex data platform and pipeline issues and drive permanent solutions.

- Collaborate with software engineers, solution architects, database architects, data scientists, analysts, and business stakeholders.

- Mentor data engineers and provide technical guidance on architecture, coding standards, design reviews, and engineering best practices.

- Review technical designs and code to ensure adherence to scalability, reliability, security, and maintainability standards.

- Drive continuous improvement and modernization of existing data platforms and engineering processes.

- Ensure technical documentation, architecture decisions, data flows, and operational processes are maintained effectively.