Data Engineer II

• Lead the design and development of data pipelines for seamless integration of data from various sources into the G2 Data Platform.

• Optimize data pipelines, ensuring cost effectiveness, scalability, and reliability.

• Constantly innovate to make the data stack follow the latest trends.

• Service data requests from various users of the G2 data platform.

• Actively contribute to data modeling and design reviews, striving for improved adoption and efficiency.

• Execute the project tasks aligned with project timelines and objectives under the guidance of senior team members.

• Develop repeatable and scalable code that processes data to ensure data availability in the platform is as real-time as possible.

• Actively contribute to the development and advancement of the data platform.

• Promote architectural changes that increase the scalability of our data infrastructure while maintaining efficiency in all phases of the development lifecycle.

Data quality assurance and governance :

• Learn and adopt best practices in data engineering to contribute to robust solutions.

• Own the implementation of data quality and data governance initiatives and drive them to completion.

• Recommend and ensure the data platform follows privacy and security standards and requirements.

• Document data architecture, data model, and workflows for how it should be consumed.

Mentorship and Collaboration :

• Guide junior engineers by providing technical support, expertise, best practices, and constructive feedback on data engineering techniques.

• Collaborate with peers, actively participating in knowledge sharing sessions and contributing to a collaborative team environment.

• Seek guidance and mentorship from senior team members to enhance technical and analytical skills.

Minimum Qualifications:

We realize applying for jobs can feel daunting at times. Even if you don’t check all the boxes in the job description, we encourage you to apply anyway. 

•  4+ years of experience as a data engineer or ETL developer.

• 2+ years of development experience with sound skills in data modeling, optimization, and database architecture.

• Experience in the design and development of data pipelines using cloud and open-source tools.

• Proficiency in writing and debugging SQL queries.

• Good programming skills in Python or Java.

• Must have good knowledge of performance tuning, optimization and debugging of data pipelines.

• Working knowledge of the AWS data services (DynamoDB, RDS, Data Pipeline, EMR, Lambda, Glue, ECS, etc.) and cloud data warehouses like SnowFlake.

• Proficiency in handling structured and unstructured data.

• Proficiency in ELT/ELT tools like AWS Glue, Step Functions, Data Pipelines, Airflow, Airbyte, and DBT.

• Familiarity with distributed computing and frameworks like Apache Spark, Hadoop, and Apache Kafka for handling large volumes of data.

• Familiarity with software engineering principles and best practices.

What Can Help Your Application Stand Out:

• Experience with Docker and Kubernetes.

• Proficiency in data modeling, schema design, and optimizing data structures for performance in Snowflake.

• Working experience in startup environments.

• Experience with Agile process methodology, CI/CD automation, Test Driven Development.

• Knowledge of data governance, security, and compliance standards within cloud-based data solutions.

• Understanding of any reporting tools such as Tableau, Qlikview ,Looker, PowerBI, etc.

  • Database administration background.