Data Engineer II

• Construct intricate ETL pipelines utilizing Datanet (Redshift) and Craddle (Spark Sql ).
• Create and manage datasets within AWS Dynamo DB, AWS S3, and Amazon Redshift.
• Develop complex ETL workflows employing AWS Glue.
• Orchestrate step functions through Python / Scala.
• Implement and enforce rigorous data quality standards and data governance policies, ensuring data accuracy, consistency, and security. Establish procedures for data validation, cleansing, and lineage tracking.
• Continuously monitor and optimize data pipelines and processes to enhance performance and efficiency. Identify bottlenecks and implement enhancements to expedite data processing and minimize latency.
• Collaborate closely with data analysts and visualization experts to facilitate data-driven decision-making. This includes designing and maintaining data reporting and visualization solutions.
• Demonstrate profound expertise in AWS data services, including Amazon S3, Glue, EMR, Redshift, Athena, and more. Select the most appropriate services for specific data engineering tasks and leverage their capabilities effectively.
• Develop and maintain mission-critical applications extensively used by sellers worldwide.
• Have good knowledge on scripting using either Python or Scala