A successful candidate will have a passion for clarity, be a self-starter comfortable with ambiguity, have strong attention to detail, be able to work in a fast-paced and entrepreneurial environment and be driven by a desire to innovate. Given the complex nature of the questions we are working on, the ideal candidate will be relentlessly curious and be able to/be interested in having the business conversation underlying the data.
Key job responsibilities
Develop novel scientific methodologies for evaluating autonomous, tool-using agents, advancing the state of the art in agent behavioral measurement and developer-experience research.
Design rigorous, reproducible experiments that yield causal insight into agent decision-making, controlling for confounds and ensuring findings are grounded in observed behavior rather than self-report.
Apply deep knowledge of cloud architecture and services to reason about the architectural decisions agents make — the service trade-offs, integration constraints, and deployment considerations that shape what gets built.
Develop verification and synthesis methods that ensure only evidence-grounded conclusions survive, advancing techniques for reliability and faithfulness in agent evaluation.
Design evaluation methodologies and metrics that quantify the quality of agent-produced outcomes across dimensions such as correctness, robustness, and security posture.
Publish and present your work at internal and external scientific venues in the fields of ML.
Collaborate with scientists, engineers, and product managers to design and implement science-focused systems.
Proficiency in model development, validation and implementation for NLP applications.
Excellent verbal and written communication and presentation skills, ability to convey rigorous mathematical concepts and considerations to non-experts.
A day in the life
As a scientist on the team, you will be involved in every aspect of the process from idea generation, business analysis and scientific research, through development and deployment of advanced models and agentic systems, giving you a real sense of ownership. You'll research, design, and build new methodologies to evaluate the experience of autonomous agents building on AWS — developing the measurement science that lets us reason rigorously about what agents do and why. Besides theoretical analysis and innovation, you will work closely with talented engineers and put your algorithms and models into practice.
About the team
Customer Experience and Business Trends (CXBT) is an organization made up of a diverse suite of functions dedicated to deeply understanding and improving customer experience, globally. We are a team of builders that develop products, services, ideas, and various ways of leveraging data to influence product and service offerings – for almost every business at Amazon – for every customer (e.g., consumers, developers, sellers/brands, employees, investors, streamers, gamers).
Our approach is based on determining the customer need, along with problem solving, and we work backwards from there. We use technical and non-technical approaches and stay aware of industry and business trends. We are a global team, made up of a diverse set of profiles, skills, and backgrounds – including: Scientists, Product Managers, Software Developers, Computer Vision experts, Solutions Architects, Business Intelligence Engineers, Business Analysts, Risk Managers, and more.
3+ years of building models for business application experience
PhD, or Master's degree and 4+ years of CS, CE, ML or related field experience
Experience in patents or publications at top-tier peer-reviewed conferences or journals
Experience programming in Java, C++, Python or related language
Experience in any of the following areas: algorithms and data structures, parsing, numerical optimization, data mining, parallel and distributed computing, high-performance computing
Hands-on experience (academic or industrial) building ML models for natural language processing tasks using GenAI
Deep working knowledge of cloud architecture and services (e.g., compute, storage, databases, auth, networking) and the trade-offs involved in composing them into production systems
Experience with agentic/tool-using systems, LLM-based agents, or the evaluation of autonomous agents
Experience using cloud-based services and big data processing technologies such as AWS Redshift, EMR, Spark, and Sagemaker
Prior work experience as a research scientist or applied scientist at a consumer product company
Excellent problem-solving ability. Demonstrable track record of dealing well with ambiguity, prioritizing needs, and delivering results in a dynamic environment
Experience in professional software development
Amazon is an equal opportunity employer and does not discriminate on the basis of protected veteran status, disability, or other legally protected status.
Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process, including support for the interview or onboarding process, please visit https://amazon.jobs/content/en/how-we-hire/accommodations for more information. If the country/region you’re applying in isn’t listed, please contact your Recruiting Partner.
The base salary range for this position is listed below. Your Amazon package will include sign-on payments and restricted stock units (RSUs). Final compensation will be determined based on factors including experience, qualifications, and location. Amazon also offers comprehensive benefits including health insurance (medical, dental, vision, prescription, Basic Life & AD&D insurance and option for Supplemental life plans, EAP, Mental Health Support, Medical Advice Line, Flexible Spending Accounts, Adoption and Surrogacy Reimbursement coverage), 401(k) matching, paid time off, and parental leave. Learn more about our benefits at https://amazon.jobs/en/benefits.
USA, WA, Seattle - 142,800.00 - 193,200.00 USD annually