Data Engineer

• Display technical expertise in data analytics focusing on a team of diversified technical competencies.
• Build and maintain accurate and scalable data pipeline and infrastructure such as SQL Warehouse, Data Lakes, etc. using Cloud platforms (e.g.: MS Azure, Databricks).
• Proactively work with business stakeholders to understand data lineage, definitions, and methods of data extraction.
• Write production-grade SQL and PySpark code to create data architecture.
• Consolidate SQL databases from multiple sources, data cleaning, and manipulation in preparation for analytics and machine learning.
• Use data visualization tools such as Power BI to create professional quality dashboards and reports.
• Write good quality documentation for data processing for different projects to ensure reproducibility.
• Living Hitachi Energy’s core values safety and integrity, which means taking responsibility for your own actions while caring for your colleagues and the business.

Your Background:

• BE / B.Tech in Computer Science, Data Science, or related discipline and at least 5 years of related working experience.
• 5 years of data engineering experience, with understanding lake house architecture, data integration framework, ETL/ELT pipeline, orchestration/monitoring, star schema data modeling.
• 5 years of experience with Python/PySpark and SQL(Proficient in PySpark, Python, and Spark SQL)
  • 2-3 years of hands-on data engineering experience using Databricks as the main tool (meaning >60% of their time is using Databricks instead of just occasionally).
    • 2-3 years of hands-on experience with different Databricks components (DLT, workflow, Unity catalog, SQL warehouse, CI/CD) in addition to using notebooks.