Starting January 26, 2026, Microsoft AI (MAI) employees who live within a 50- mile commute of a designated Microsoft office in the U.S. or 25-mile commute of a non-U.S., country-specific location are expected to work from the office at least four days per week. This expectation is subject to local law and may vary by jurisdiction.
Responsibilities
Define the future of quality measurement
Own the long-term vision and roadmap for Bing's offline quality metrics ecosystem.
Define measurement strategies for AI-powered search, answer experiences, ranking systems, retrieval, multimodal content, and agentic experiences.
Identify gaps in existing quality measurements and drive invention of next-generation metrics.
Lead metric design and validation
Partner with Data Scientists, Applied Scientists, and Research teams to create robust, trustworthy, and actionable quality metrics.
Drive metric validation, calibration, and governance frameworks.
Ensure offline metrics accurately predict user value and align with online outcomes.
Build organizational alignment
Drive alignment across Search, Copilot, Answer Experiences, Ranking, Ads, Multimedia, and other partner teams.
Influence product roadmaps through measurement-based insights.
Serve as the primary point of contact for quality measurement across multiple organizations.
Turn measurement into product insight
Move the organization from reporting scores to explaining movement. Build diagnostic frameworks that surface which scenarios, segments, and page configurations drive competitive gaps, and translate those into concrete investment opportunities.
Enable product decision-making
Ensure teams can confidently use measurement systems for product development, experiment evaluation, launch readiness, executive scorecards, competitive tracking, and quality monitoring.
Translate complex evaluation concepts into actionable recommendations for engineering and leadership teams.
Shape the industry
Stay at the forefront of advances in search evaluation, LLM evaluation, AI quality measurement, human-in-the-loop evaluation, and synthetic evaluation approaches.
Opportunity to influence Microsoft's overall approach to AI quality measurement.
Qualifications
Required Qualifications
Bachelor's Degree AND 6+ years experience in engineering, product/technical program management, data analysis, or product development
OR equivalent experience.
3+ years of experience managing cross-functional and/or cross-team projects.
Preferred Qualifications
Bachelor's Degree AND 12+ years experience engineering, product/technical program management, data analysis, or product development
OR equivalent experience.
Experience defining or operating metrics used for large-scale product decisions.
Demonstrated ability to lead cross-functional initiatives across multiple teams.
Solid analytical and quantitative problem-solving skills.
Excellent communication and stakeholder management abilities.
Experience in search, recommendation systems, ranking, AI products, or large-scale consumer experiences.
Experience building evaluation frameworks for LLMs, retrieval systems, or generative AI applications.
Familiarity with offline evaluation methodologies and human rating systems
Experience partnering with Data Science, Applied Science, and Research organizations.
Proven success driving strategy in ambiguous and rapidly evolving technical domains.
#AIX
#metrics
#bing
#search
#quality
#relevance
Technical Program Management IC5 - The typical base pay range for this role across the U.S. is USD $142,800 - $274,800 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $188,000 - $304,200 per year.
- Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here:
- https://careers.microsoft.com/us/en/us-corporate-pay
This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled.
Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.
