Primary responsibilities
Own Evaluation and Quality workflows
Own the roadmap for shared agent evaluation and observability capabilities, from user discovery and requirements through launch, adoption and improvement.
Partner with domain owners to define representative journeys, prompt sets, expected tool behavior and outputs, scoring rubrics and acceptance criteria.
Define requirements for automated evaluation, human review, and tools that help teams diagnose failures.
Connect Evaluations with Operational Evidence
Define product requirements for traces, events, diagnostic views and review tools that connect customer intent, tool or API calls and observed outcomes where data access permits.
Work with engineering, analytics and existing enterprise evaluation and observability teams to extend shared capabilities, reuse instrumentation and standardize evidence.
Identify where limited access to partner data prevents a complete assessment.
Partner with Identity and domain owners to test policy-relevant behavior, including denied actions, revoked access and required approvals.
Measure and Improve Agent Experiences
Define meaningful measures such as task completion, correct tool use, response quality, failure and recovery rates, latency and cost.
Define how testing and monitoring work together across controlled test environments, end-to-end partner integrations and production. Clarify what each can measure and use the findings to prioritize improvements with the responsible teams.
Measure adoption and usefulness of evaluation workflows, including review effort, coverage and time to identify problems. Communicate the limits of findings based on sampled data.
Lead a Reusable Product Capability
Translate measurement gaps into a prioritized backlog, clear requirements, and actionable acceptance criteria. Recommend when to extend existing tools, integrate a solution or build a missing capability.
Build privacy and data governance into evaluation and telemetry requirements, including permitted uses of partner-provided data, appropriate access, minimization and retention. Communicate uncertainty, release risks and tradeoffs clearly.
Minimum Qualifications
6-8+ years of experience in product management, product development, software engineering, technical platforms, analytics, quality tooling, or AI products.
Experience owning evaluation, testing, observability, analytics or related technical products through launch, adoption and iteration, with measurable user or business outcomes.
Practical understanding of AI or agent-enabled systems, including nondeterministic behavior, tool or API interactions, and the need for both automated tests and human judgment.
Ability to define metrics, interpret evidence and identify sampling, coverage, or measurement limitations. Familiarity with traces, logs, APIs and technical troubleshooting.
Strong product discovery, roadmap and prioritization skills, including translating ambiguous quality problems into usable tools and repeatable workflows.
Cross-functional leadership and clear communication, with the ability to balance customer outcomes, engineering constraints and privacy or risk requirements.
Preferred Qualifications
Experience with LLM or agent evaluation, scenario design, human review, model-assisted scoring or production failure analysis.
Familiarity with MCP, function calling, telemetry standards or evaluation frameworks; ability to use code or queries to investigate evidence.
Experience building shared capabilities adopted by multiple teams or delivering products in financial services or another regulated industry.
Bachelor’s degree or equivalent experience.
Employment eligibility to work with American Express in the United States is required as the company will not pursue visa sponsorship for these positions.