Databricks Architect
willwaretech
Job Description
- Design and implement end-to-end Lakehouse Architecture using Databricks and Delta Lake.
- Develop and maintain Medallion Architecture with Bronze, Silver, and Gold data layers.
- Lead data governance, access control, security, cataloging, and lineage using Unity Catalog.
- Optimize Apache Spark workloads, cluster configurations, Delta Live Tables (DLT), and Databricks SQL queries.
- Build scalable ETL/ELT pipelines using PySpark, Spark SQL, and Python.
- Establish CI/CD pipelines and development best practices using Git and tools such as Azure DevOps or GitHub Actions.
- Integrate Databricks with cloud data platforms such as Azure, AWS, or GCP.
- Implement orchestration using Azure Data Factory, Airflow, or similar tools.
- Translate business requirements into technical architecture, data models, and implementation plans.
- Mentor data engineering teams and drive coding standards, data quality, reliability, and cost optimization.