Senior Data Engineer

Primary Responsibilities

• Design and develop scalable data pipelines across the full data lifecycle—from ingestion and transformation to consumption.
• Work with large-scale data volumes (TBs/PBs) and ensure efficient data processing and storage.
• Collaborate with business stakeholders to identify high-impact problems and translate them into data solutions.
• Apply advanced knowledge of SQL, Spark, Python, Scala, and the Hadoop ecosystem.
• Implement and manage Data Lake/Lakehouse architectures using platforms like Cloudera, Hortonworks, and AWS.
• Develop APIs and integrate them into data workflows for seamless data access and usability.
• Utilize big data infrastructure tools including Spark Streaming, EMR, MapReduce, Hive, HDFS, YARN, HBase, and Oozie.
• Create and maintain technical design documentation (HLD/LLD) for data pipelines and architecture.
• Debug and resolve technical issues in data architecture and ensure high availability and performance.

Secondary Responsibilities

• Work independently and manage your own development efforts.
• Collaborate with BI analysts and engineers to prototype and implement analytical models.
• Identify key performance indicators and define strategies to meet analytical goals.
• Present data solutions and insights to stakeholders and leadership teams.

Managerial & Leadership Responsibilities

• Lead moderately complex data engineering initiatives aligned with enterprise strategy.
• Build and maintain optimized, highly available data pipelines to support analytics and reporting.
• Oversee data integration efforts, including modeling, warehousing, and analytics environments.
• Resolve complex data engineering issues while ensuring compliance with data governance policies.
  • Collaborate with peers and managers to align on strategic goals and deliverables.