Data/ETL Engineer

Technical Design

• Develop and maintain data pipelines and ETL workflows using Databricks and Airflow.
• Good understanding of Databricks Delta Live tables, and Unity Catalog
• Build frameworks and templates for ETL/ELT, for telemetry pipelines.
• Ensure cloud-native designs aligned with AWS platform best practices.

Governance, Security & Optimization

• Ensure data quality, integrity, and governance across data platforms.
• Optimize SQL queries and data models for performance and scalability.
• Implement monitoring and observability for data workflows.

Cross-Functional Collaboration

• Work closely with architects and leads to understand data requirements and build technical solutions.
• Collaborate with analytics and business teams to deliver actionable insights through BI dashboards and reports

AI/LLM Enablement (Advantage)

• Familiarity with Agentic AI concepts for automation, data quality validation, and metadata enrichment will be an added advantage.
•  

Required Skills & Experience

• 8+years of experience in data engineering, with at least 3 years in Databricks.
• Hands-on expertise in:
• Databricks (PySpark, Spark SQL, Delta Lake, Unity Catalog, Delta Live tables)
• SQL for data modeling and query optimization
• AWS services (S3, Glue, Kinesis, Redshift, API Gateway, SNS, Lambda)
• Airflow for workflow orchestration
• Added advantage – DBT, Splunk, Atlan, Java, API, Power BI
  • Strong analytical and problem-solving skills with attention to detail.