Data Engineer III
deloitte
Job Description
- 6 - 10 years of hands-on experience in Apache Spark and Python programming
- Deep technical understanding of distributed computing and broader awareness of different Spark versions
- Strong UNIX operating system concepts and shell scripting knowledge
- Hands-on experience using Spark & Python
- Deep experience in developing data processing tasks using PySpark such as reading data from external sources, merging data, performing data enrichment and loading in to target data destinations.
- Experience in deployment and operationalizing the code, knowledge of scheduling tools like Airflow, Control-M etc. is preferred
- Working experience on AWS ecosystem, Google Cloud, BigQuery etc. is an added advantage
- Hands on experience with AWS S3 Filesystem operations
- Good knowledge of Hadoop, Hive and Cloudera/ Hortonworks Data Platform
- Should have exposure with Jenkins or equivalent CICD tool & Git repository
- Experience handling CDC operations with a huge volume of data
- Should understand and have operating experience with Agile delivery model
- Should have experience in Spark related performance tuning
- Should be well versed with understanding of design documents like HLD, TDD etc
- Should be well versed with Data historical load and overall Framework concepts
- Should have participated in different kinds of testing like Unit Testing, System Testing, User Acceptance Testing, etc