Data Engineer

• Building and optimizing data pipelines, data warehouses, and data lakes on the AWS and Databricks platforms.

• Managing and maintaining the AWS and Databricks environments.

• Ensuring data integrity, accuracy, and consistency through rigorous quality checks and monitoring.

• Maintain system uptime and optimal performance

• Working closely with cross-functional teams to understand business requirements and translate them into technical solutions.

• Exploringand implementing new tools and technologies to enhance ETL platform performance.

Functional Skills:

Must-Have Skills:

• Proficientin SQL for extracting, transforming, and analyzing complex datasets from both relational and columnar data stores. Proven ability to optimize query performance onbig dataplatforms.

• Proficientin leveraging Python,PySpark, and Airflow to build scalable and efficient data ingestion, transformation, and loading processes.

• Ability to learn new technologies quickly. Strong problem-solving and analytical skills. Excellent communication and teamwork skills.

Good-to-Have Skills:

• ExperiencedwithSQL/NOSQL database, vector database for large language models

• Experiencedwith data modeling and performance tuning for both OLAP and OLTP databases 

• Experiencedwith Apache Spark, Apache Airflow

• Experiencedwith software engineering best-practices, including but not limited to version control (Git, Subversion, etc.), CI/CD (Jenkins, Maven etc.), automated unit testing, and Dev Ops

• Experiencedwith AWS, GCP or Azure cloud services