Databricks+spark

• Design robust Databricks based data platform architectures that enable scalable analytics and reporting for complex enterprise use cases and align with long term business growth objectives.
• Drive end to end solution design using Databricks SQL and PySpark to optimize data ingestion transformation and consumption flows that deliver consistent performance for diverse workloads.
• Define standardized patterns and reusable components for Databricks Workflows that improve orchestration reliability simplify maintenance and reduce operational overhead across multiple projects.
• Collaborate with data engineers analysts and product teams in a hybrid work environment to translate business needs into technical blueprints that ensure clear traceability from requirement to implementation.
• Establish governance practices for data quality security and compliance on Databricks platforms that support responsible data usage and protect sensitive information across regions.
• Optimize SQL queries PySpark jobs and cluster configurations to reduce processing time and cost while maintaining high availability and resilience for critical data applications.
• Guide teams on best practices for version control environment management and deployment pipelines so that Databricks solutions are consistently delivered with predictable quality.
• Review solution designs and technical deliverables for complex data initiatives to identify risks propose improvements and ensure alignment with architectural standards and organizational guidelines.
• Coordinate with infrastructure and cloud operations stakeholders to ensure Databricks environments are correctly sized monitored and tuned for day shift usage patterns without requiring travel.
• Promote data engineering excellence through documentation technical knowledge sharing and coaching that help colleagues build stronger skills in Databricks SQL Workflows and PySpark.
• Engage with business sponsors to explain architectural decisions in clear terms demonstrating how the data platform improves decision making and contributes to broader societal value through better insights.
• Assess new features and capabilities in the Databricks ecosystem and recommend pragmatic adoption strategies that balance innovation stability and regulatory considerations.
• Monitor production platforms and incident trends to drive continuous improvement initiatives that enhance reliability reduce defects and protect the organization reputation.

Qualifications

• Possess extensive experience of twelve to sixteen years in data architecture or advanced data engineering roles with a strong focus on modern cloud based analytic platforms.
• Demonstrate expert level proficiency in Databricks SQL including complex query design performance tuning and implementation of data models for large scale analytical workloads.
• Show deep hands on capability in PySpark for building scalable batch and streaming data pipelines that integrate disparate data sources into unified and well structured datasets.
• Hold strong practical experience designing and managing Databricks Workflows including job orchestration scheduling dependency management and error handling for business critical processes.
• Exhibit solid understanding of data warehousing lakehouse concepts and distributed computing principles that underpin resilient architectures for enterprise analytics.
• Display experience working in hybrid work models with global teams using collaborative tools and structured communication practices that keep projects aligned and efficient.
• Bring familiarity with cloud security data governance and regulatory considerations so that architecture decisions safeguard data assets and meet organizational policies.
• Prefer exposure to adjacent tools such as data catalog solutions and mainstream visualization platforms that consume outputs produced from Databricks environments.
  • Value clear documentation architectural diagrams and concise technical narratives that improve knowledge transfer and reduce onboarding time for new project members.