AI Ops Engineer

• Gate new AI agents, prompts, and models before production release
• Design and run evaluation frameworks using deterministic testing and LLM-as-a-Judge methodologies
• Build and maintain golden datasets, annotation pipelines, and human-review workflows
• Monitor AI quality, hallucinations, drift, latency, cost, and safety metrics
• Convert customer feedback and production learnings into measurable quality improvements
• Drive AI adoption and quality standards across your assigned product portfolio

What We're Looking For

• 1-3 years of experience in AI Engineering, AI Operations, AI Quality Engineering, LLMOps, MLOps, Test Automation, or related disciplines

• Experience with Langfuse, LangSmith, Phoenix, or similar LLM observability/evaluation platforms
• Strong understanding of LLMOps, AI quality engineering, and evaluation frameworks
• Experience with hallucination detection, model drift monitoring, prompt regression management, and AI safety evaluations
• Hands-on experience building evaluation datasets, quality benchmarks, and feedback loops
• Familiarity with NLP concepts such as embeddings, semantic similarity, classification, and retrieval
  • Strong product mindset and stakeholder management skills