Evaluation, safety, and quality
Build and maintain the evaluation harness for your agents: gold sets, offline benchmarks, LLM-as-judge with human calibration, online A/B, regression gates in CI/CD.
Instrument every agent with business KPIs (deflection rate, time-to-resolve, CSAT, cost per interaction) and quality KPIs (task success, groundedness, refusal correctness, safety incidents).
Partner with cyber, privacy, and model risk to make guardrails concrete: PII handling, prompt-injection & jailbreak defense, tool-use hygiene, secret handling, prompt/model version pinning.
Production reliability (SRE for agents)
Own SLOs and error budgets for your agents; write runbooks; take part in the on-call rotation.
Build canaries, circuit breakers, deterministic replay, and rollback paths for prompts, models, and tools.
Diagnose and fix production issues end-to-end — from token spend anomalies to tool call failures to hallucination regressions.
Platform contribution
Contribute to the shared agentic platform (orchestration, model gateway, vector/knowledge stores, telemetry, eval harness) — not just consume it. Push improvements upstream so other squads benefit.
Write design docs, patterns, and internal blog posts. Represent the team in architecture reviews.
Technical leadership without being a manager
Set the technical bar in your squad through code reviews, pairing, and design reviews.
Mentor mid-level and junior engineers; grow the next generation of AI engineers.
Interview candidates and help sustain a strong hiring bar.
Required qualifications, capabilities and skills
8+ years of software engineering experience, with 2+ years shipping production LLM- or agent-based systems to real users (not just prototypes or notebooks).
Deep hands-on expertise in Python (primary) and comfort with at least one other modern language (TypeScript / Java / Go).
Practical, current experience with the agentic stack: LLM orchestration (e.g. LangGraph, LlamaIndex, Semantic Kernel, custom), tool/function calling, RAG, evaluation harnesses, guardrails, prompt & model versioning, model gateways.
Hands-on with Model Context Protocol (MCP) architectures — designing servers/clients, tool schemas, capability negotiation, and secure enterprise deployment patterns.
Track record of building and running production-grade backend / distributed systems (APIs, queues, workers, observability, on-call).
Strong grounding in data platforms (Databricks / Snowflake / lakehouse patterns) and modern MLOps/LLMOps (CI/CD, feature/prompt/model registry, telemetry).
Practical experience with responsible AI, security, and controls for enterprise deployments: PII handling, red-teaming, prompt-injection defense, secret & tool-use hygiene.
Clear technical communication: you can write a crisp design doc, review a PR with substance, and explain a trade-off to a non-technical stakeholder.
Bachelor's in Computer Science or related discipline, or equivalent industry experience.
Preferred qualifications, capabilities and skills
Experience building employee-facing / workforce productivity products at scale (IT service desk, HR tech, knowledge management, workplace assistants).
Experience integrating with ServiceNow, identity providers, endpoint management, and enterprise search.
Experience with multi-agent patterns (planner-executor, supervisor-worker, debate/critique) and long-horizon workflows.
Contributions to open source (especially in the MCP / agent / eval ecosystem), published research, patents, or public talks in AI / agents / SRE.
Prior experience in a highly regulated environment (financial services, healthcare, public sector) with model risk management (SR 11-7 or equivalent).
Master's or PhD in a quantitative discipline.