Data Engineer II

• Pipeline reliability — exactly-once Flink enrichment, at-least-once Faust analytics, zero data loss at high throughput
• ClickHouse schema and tiering — DDL, materialized views, TTL policies, async-insert tuning, ingestion watermarks and reconciliation jobs
• Faust state management — RocksDB-backed per-entity state (Kalman, anomaly hot-windows), partition rebalancing across deployment tiers
• Redpanda operations — topic design, partition strategy, retention, compaction, consumer-group lag monitoring and backpressure admission control
• Observability — trace_id propagation, latency histograms, DLQ routing, consumer-lag HPA metrics
• Data supply chain — offline SBOM, air-gapped package verification, immutable reference data deployment

Must-Have Requirements (3+ years)

• Kafka or Kafka-equivalent streaming backbone experience — any production variant (Apache Kafka, Confluent, Redpanda, etc.)
• Microservice-based ETL on streaming data — must have built stream processing logic inside microservices, not through SQL queries or orchestration-only tools
• Production streaming data pipeline experience — hands-on experience on a live, high-velocity, continuous time-series data pipeline at scale

Good to Have

• Apache Flink (Java/DataStream API) — checkpointing, stateful processing, exactly-once sinks
• Faust (Python) or Kafka Streams — stateful stream processing, worker scaling
• ClickHouse or any columnar/OLAP database at scale (Redshift, Aurora, TimescaleDB also acceptable)
• Air-gapped, bare-metal, or regulated environment experience
• Multi-language proficiency across Python, Java, and Scala simultaneously
• Technical mentorship or team lead experience
• CMMI Level 5 / ISO 25010 documentation discipline — ADRs, SDD sections, RTM traceability

Non-Negotiables

• No cloud-only experience — must be comfortable operating self-managed infrastructure
• Comfortable writing design pseudo-code (Python/Java/SQL) that becomes implementation
  • Documentation-as-code mindset