Lead human and automated evaluation. Establish reviewer guidelines, calibrate LLM-as-a-judge systems against expert assessments, and measure agreement, false positives, false negatives, and severity classification.
Drive model fitness decisions. Collaborating with LLMOpn in the Localization Technology team to compare models and configurations by locale, task, quality, cost, and latency to inform model selection, routing, and deployment.
Close the quality improvement loop. Translate findings into prioritized improvements to prompts, retrieval, terminology, workflows, and model configurations. Partner with owners to verify fixes and prevent regressions.
Operationalize continuous quality monitoring. Establish regression testing, production sampling, quality dashboards, escalation processes, and ongoing evaluation as models and user needs evolve.
Lead cross-functional execution. Clarify ownership, resolve dependencies, communicate risks, and align stakeholders on quality tradeoffs and launch readiness.
Basic Qualifications
Minimum 5 years of professional experience in program management within a high-growth technology or large-scale enterprise environment.
Bachelor’s degree in Business Operations, Engineering, or a related field; or equivalent professional experience.
Preferred Qualifications
8+ years of professional experience in program management within a high-growth technology or large-scale enterprise environment.
Experience leading complex technical programs involving AI/ML, software platforms, quality engineering, or data-driven products.
Demonstrated ability to deliver cross-functional programs with Engineering, Localization team, Data Scientist, Product, and Operations.
Working understanding of LLM capabilities and limitations, including hallucinations, prompt engineering, prompt injection, RAG, context management, and tool use.
Experience designing quality metrics, evaluation methodologies, human-review workflows, or experimental programs.
Ability to interpret statistical results, assess dataset quality, and communicate the limitations of evaluation findings to leaderships and LLMOps.
Strong analytical, stakeholder management, and written communication skills, with the ability to turn ambiguous requirements into executable plans.
Experience evaluating multilingual LLMs, machine translation, conversational AI, or localization quality.
Familiarity with MQM-based evaluation, terminology governance, and market-specific linguistic requirements.
Experience with LLM evaluation platforms, automated regression pipelines, and production observability.
Familiarity with SQL, Python, APIs, and data pipelines sufficient to investigate results and collaborate effectively with technical teams.
Experience evaluating multimodal systems, including text, image, audio, or video.
Experience establishing quality standards and launch criteria for AI systems operating at scale.
For Sunnyvale/San Francisco, CA-based roles: The base salary range for this role is USD $167,000 per year - USD $185,000 per year.
You will be eligible to participate in Uber's bonus program, and may be offered an equity award & other types of comp. All full-time employees are eligible to participate in a 401(k) plan. You will also be eligible for various benefits.