Data Engineer
compoundexpress
Job Description
- Assemble large, complex data sets from distinct sources into Data Lake that meet functional / non-functional business requirements.
- Identify, design, and implement data and ETL pipeline - optimizing data delivery, re-designing infrastructure for greater scalability, etc for both Stream and Batch data processing
- Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL, Spark, Python, Airflow and AWS ‘big data’ technologies.
- Keep our data separated and secure by adherence to infosec and regulatory guidelines related to data centers, governance, monitoring usage and selection of AWS regions.
- Create data & analytics tools/platform that assist Data & Analytics team members in building and optimizing our product into an innovative industry leader
- Expected OUTCOME : Deliver high quality Data Engineering Pipeline and Solution Architecture based on the business requirement which is scalable and streamlined
- Expected OUTCOME : Own data quality, correctness, validation, consistency & Reliability across pipeline
- Expected OUTCOME : Optimization of infrastructure based on usage and requirement
- Expected OUTCOME : Documentation and maintenance of Production code for efficient root cause analysis in case of any failures/issues