Responsibilities
Build trusted measures of quality
Develop online and offline metrics, representative evaluation datasets, human and AI-assisted labeling approaches, and benchmarks grounded in real customer tasks.
Validate that quality measures are statistically sound, sensitive to meaningful changes, and predictive of customer outcomes.
Turn signals into improvements
Identify high-impact failure modes across Copilot experiences and use data, evaluations, and customer feedback to understand their root causes.
Improve quality through hill-climbing approaches such as post-training, retrieval and ranking, prompting, orchestration, and product changes.
Design experiments that distinguish promising offline results from improvements customers actually experience.
Experimentation and Release Quality
Establish rigorous A/B experimentation, quality gates, regression protection, and rollout criteria that connect offline gains with improvements in satisfaction, task success, engagement, retention, latency, reliability, safety, and cost.
Create operating mechanisms that rapidly resolve quality tradeoffs, dependencies, and release risks.
Qualifications
Required Qualifications
Bachelor's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 8+ years related experience (e.g., statistics, predictive analytics, research)
OR Master's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 6+ years related experience (e.g., statistics, predictive analytics, research)
OR Doctorate in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 5+ years related experience (e.g., statistics, predictive analytics, research)
OR equivalent experience.
Preferred Qualifications
Advanced degree (MS or PhD) in a relevant technical discipline.
Experience improving large language models, AI assistants, agents, or other large-scale AI products.
Ability to operate in a fast‑paced, research‑driven environment while maintaining high technical standards.
Passion for advancing human‑aligned AI and staying at the forefront of emerging model‑training technologies.
Demonstrated commitment to inclusive leadership, talent development, and fostering a culture where teams do the best work of their careers.
Significant experience building, evaluating, or improving production ML or AI systems.
Demonstrated expertise in at least one relevant area, such as large-model evaluation, deep learning, post-training, reward modeling, reinforcement learning, retrieval, or ranking.
Experience designing rigorous metrics or experiments and using results to guide technical and product decisions.
#CAP #Copilot
Applied Sciences IC6 - The typical base pay range for this role across the U.S. is USD $165,600 - $296,400 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $220,800 - $331,200 per year.
- Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here:
- https://careers.microsoft.com/us/en/us-corporate-pay
This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled.
Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.