Senior Data Engineer

• Lead the design and development of scalable batch and real-time ETL/ELT pipelines. 

• Own complex data solutions from requirements and design through deployment and production support. 

• Build cloud-based lakehouse and data-platform solutions using Databricks and AWS. 

• Develop reusable, metadata-driven data integration frameworks. 

• Integrate structured, semi-structured, and unstructured data from enterprise, manufacturing, API, and third-party sources. 

• Optimize Spark workloads, Databricks compute, SQL queries, partitioning, storage, and caching for performance and cost. 

• Implement workflow orchestration, monitoring, alerting, data-quality controls, and recovery processes. 

• Develop CI/CD pipelines and automated testing for data solutions. 

• Implement metadata management, lineage, cataloging, governance, RBAC, and data-security controls. 

• Define data models, data contracts, integration patterns, and reusable engineering standards. 

• Lead architecture reviews, code reviews, troubleshooting, and root-cause analysis. 

• Mentor engineers and provide technical guidance across delivery teams. 

• Collaborate with architects, analysts, data scientists, product teams, and DevOps teams. 

• Support estimation, sprint planning, technical roadmaps, and delivery-risk management. 

• Evaluate emerging technologies and recommend solutions based on scalability, security, maintainability, and business value. 

• Participate in operational support, including occasional off-hours support. 

Basic Qualifications

One of the following: 

• Master’s degree in Computer Science, Engineering, Information Technology, Data Science, or a related field and at least 7 years of relevant experience. 

OR 

• Bachelor’s degree in a related field and at least 9 years of relevant experience. 

Must-Have Skills

• Advanced hands-on experience with Databricks, Apache Spark, PySpark, Spark SQL, Delta Lake, Python, and SQL. 

• Experience designing and operating production-grade batch and streaming pipelines. 

• Strong understanding of distributed computing, lakehouse architecture, data warehousing, and data integration. 

• Experience with Databricks Workflows or comparable orchestration tools. 

• Strong experience with AWS data, compute, storage, security, and monitoring services. 

• Experience with Spark performance tuning, cluster optimization, partitioning, and cost management. 

• Experience with Git, CI/CD, automated testing, monitoring, and production deployment. 

• Experience implementing data quality, metadata management, lineage, governance, and access controls. 

• Strong understanding of RBAC, least privilege, encryption, auditability, and regulated-data requirements. 

• Experience leading technical design, code reviews, and complex production implementations. 

• Ability to define reusable patterns, engineering standards, and development best practices. 

• Strong communication, collaboration, mentoring, and problem-solving skills. 

• Experience working in Agile or Scaled Agile delivery environments.