Design and execute linguistic test cases to validate Quick's natural language features across diverse inputs, edge cases, and user scenarios. Evaluate model outputs for accuracy, fluency, and relevance by applying linguistic expertise to identify regressions, hallucinations, and quality gaps. Develop and maintain testing guidelines and rubrics that ensure consistent, repeatable evaluation standards across annotators and test cycles. Analyze test results and report findings using Python and SQL to surface patterns, quantify defect rates, and prioritize issues for engineering teams. Collaborate with engineers, scientists, and PMs to define acceptance criteria, triage language-related bugs, and validate fixes before releases Work on high complex annotations and audits as needed Work effectively both independently and as part of a team Uphold a high bar for quality while delivering accurate work at pace Apply strong attention to detail and Critically assess data quality Manage competing tasks and adapt quickly to shifting priorities Take ownership and Exercise sound judgment in escalating Handle highly complex tasks and Scale with growing workloads About the team Diverse Experiences AWS values diverse experiences. Even if you do not meet all of the qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying. About AWS Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating — that’s why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses. Inclusive Team Culture Here at AWS, it’s in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon conferences, inspire us to never stop embracing our uniqueness. Work/Life Balance We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why we strive for flexibility as part of our working culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve in the cloud. Mentorship & Career Growth We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional. - 3+ years of working with Data & AI related technologies, including, but not limited to, AI/ML, GenAI, Analytics, Database, and/or Storage experience - Experience with Machine Learning and Large Language Model fundamentals, including architecture, training/inference lifecycles, and optimization of model execution - Completed a bachelor’s degree in engineering, Computer Science, Machine Learning, Operations Research, Data Science/ Linguistics - Knowledge of linguistic fundamentals, including handling ambiguity, natural language processing concepts, and query (Q&A) interpretation - Experience programming in Python or a related language - Proficient in SQL Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process, including support for the interview or onboarding process, please visit https://amazon.jobs/content/en/how-we-hire/accommodations for more information. If the country/region you’re applying in isn’t listed, please contact your Recruiting Partner.
