Design and own ETL/ELT pipelines (batch and streaming where required).

Drive Medallion architecture design (Bronze, Silver, Gold) with clear data contracts and quality checks.

Build scalable transformations using Spark/Databricks with performance tuning. Implement data validation frameworks, monitoring, lineage, and observability.

Handle schema evolution, incremental loads, idempotency, and orchestration patterns. Collaborate with content and application teams to define reliable data models.

GenAI Engineering

Design and improve production RAG systems: ingestion, chunking strategy, embedding pipelines, indexing, retrieval optimization.

Work with vector search systems: filtering, hybrid search, ranking improvements. Optimize prompt templates for grounding, structured outputs, and cost efficiency. Define evaluation metrics for retrieval quality, hallucination reduction, and latency. Contribute to architecture decisions around AI services and integration patterns.

Backend s Full-Stack Contributions

Design and implement backend APIs (FastAPI or similar frameworks). Build microservices to expose data products and AI capabilities.

Integrate APIs with frontend applications (basic Angular/JS understanding preferred). Work with authentication, API gateways, and service orchestration patterns.

Collaborate with frontend teams to ensure data contracts and performance alignment.

Must Have

3–5 years’ experience in Data Engineering / Backend Engineering. Strong Python and advanced SQL skills.

Hands-on Spark/Databricks experience with performance optimization.

Deep understanding of Medallion architecture and production data systems. Experience building reliable data pipelines with testing and monitoring.

Practical understanding of LLMs, RAG systems, embeddings, vector search. Experience building and deploying REST APIs.

Understanding of system design basics (scalability, reliability, trade-offs).