Data Engineer

kumaran

Hyderabad 4 Years Exp Posted 2h ago

Experience: 4–6 years in data engineering
Core Platform: Databricks with Python, Spark, Pandas (notebooks and modular code)
Orchestration: Azure Data Factory (pipelines, integration runtimes); ingest from diverse sources
Lakehouse: Delta Lake fundamentals; Medallion architecture (bronze/silver/gold) in production
Storage/SQL/Performance: Azure Data Lake Storage (ADLS); strong SQL; performance-aware design
Data Patterns: ETL/ELT; data modeling (e.g., dimensional/star schema)
DevOps & Security: CI/CD for data projects (Azure DevOps or GitHub Enterprise); familiarity with Azure Entra ID for SSO/RBAC; secure workspace/data access
Quality & Observability: Data validation/testing, code reviews, and basic monitoring/alerting for jobs/pipelines
Ways of Working: Agile/Scrum (Jira/Confluence); clear pipeline and data contract documentation
Collaboration: Effective stakeholder engagement; support/mentor junior team members; clear communication
Generative AI (Day 1):
- Prompt design for data tasks (ingestion, transformations, documentation) with clear objectives and constraints
  - Use of Copilot/ChatGPT to scaffold notebooks/jobs, generate tests, and optimize SQL/Spark—validates outputs before merging