EY - GDS Consulting - AI And DATA - Databricks Architect - Senior Manager
ey
Job Description
- Architect, design, and deliver scalable Data & AI solutions on the Databricks Lakehouse platform, covering ingestion, transformation, storage, governance, and consumption layers.
- Design and optimize large-scale distributed data processing pipelines using Hadoop, Apache Spark, and PySpark.
- Build reusable, production-grade Python frameworks and pipelines for batch and streaming data engineering workloads.
- Define and implement cloud data architecture on AWS and/or Azure, covering storage, IAM, networking, security, and monitoring, with native Databricks integration.
- Own end-to-end ETL/ELT design, data ingestion, transformation, orchestration, and data quality frameworks across the platform.
- Design dimensional data models (star/snowflake schemas, fact/dimension tables, SCD) to support analytical and reporting use cases.
- Design and build Agentic AI solutions on Databricks, including LLM integration, tool/function calling, agent workflows, guardrails, and enterprise data integration.
- Architect and implement RAG (Retrieval-Augmented Generation) solutions using Databricks — document ingestion, chunking, embeddings, vector search, retrieval, grounding, and evaluation.
- Establish and enforce engineering best practices for Git-based version control, code reviews, and CI/CD across development, test, and production environments.
- Lead performance optimization and production deployment of Databricks jobs, workflows, and pipelines at scale.
- Participate in strategic RFP/RFI responses, technical proposals, solution shaping, effort estimation, and architecture presentations.
- Act as a trusted technical advisor in customer workshops, translating business requirements into robust technical architectures.
- Mentor engineering teams on Databricks, Spark, cloud architecture, and modern AI/RAG engineering practices.