Data Engineer / Senior Data Engineer
ycombinator
Job Description
- Build and operate the data lake powering Cyble's AI agents: ingestion, Data modeling, deduplication, freshness, lineage, and retrieval-ready outputs.
- Design and run PySpark pipelines on EMR over Apache Iceberg on S3 (Glue Data Catalog), including schema evolution, partitioning, compaction, snapshot expiry, and small-file control.
- Ingest scraped, crawled, and leaked data from heterogeneous sources (HTML, JSON, CSV, text dumps, archives, feeds, APIs) and handle the realities: encoding issues, schema drift, malformed records, duplicates, and re-crawled content.
- Build and tune OpenSearch/Elasticsearch indexing pipelines: mappings, analyzers, shard and index lifecycle strategy, bulk ingestion, and query performance.
- Build the data side of RAG and agent workloads: chunking, embedding generation, vector and hybrid (keyword + kNN) indexing, and datasets the agents can trust.
- Run streaming and event-driven pipelines on Kafka; orchestrate batch workloads with Airflow.
- Deploy and operate data workloads on Kubernetes (EKS) and ECS, including Spark jobs, Trino, workers, and supporting services.
- Own lake and pipeline reliability: monitoring, alerting, data quality checks, backfills and reprocessing, retention, and storage/compute cost control.
- Use AI coding assistants and agents (Claude Code, Cursor, Copilot, or similar) daily to build, refactor, test, debug, and document, and review AI-generated code critically before it reaches production.
- Work with security researchers, data scientists, and product engineers on data models for new intelligence products.