Senior Data / ML Engineer
webspiders
Job Description
- Design, develop, and optimize large-scale ETL/ELT pipelines using AWS services such as EMR (Spark), Glue, Lambda, and Step Functions.
- Orchestrate complex data workflows with Apache Airflow (Amazon MWAA or self-managed), ensuring reliability, observability, and SLA adherence.
- Architect and manage data storage solutions across Amazon S3 (data lake), Redshift (data warehouse), and RDS (relational databases), applying best practices for partitioning, compression, and cost optimization.
- Build and maintain containerized data applications and microservices using Docker and Amazon ECS/Fargate, including CI/CD automation.
- Develop event-driven and serverless data processing solutions with AWS Lambda, SQS, SNS, and EventBridge.
- Leverage AI-powered coding assistants and IDE integrations (e.g., Kiro, Cursor, Claude Code) to accelerate development, code review, and documentation.
- Implement data quality frameworks, monitoring, and alerting to ensure data integrity across all pipelines.
- Collaborate with Data Scientists to productionize ML models and feature pipelines.
- Define and enforce data governance, security, and access-control policies in line with organizational and regulatory standards.
- Contribute to infrastructure-as-code initiatives using Terraform, CloudFormation, or CDK.