We are seeking a QA Engineer for AI Products for a six-month contract position in Bellevue, Washington. This is a hybrid role focused on testing and evaluating large language model (LLM) and generative AI features, with hands-on responsibilities across model validation, test automation, and quality assurance frameworks.
Responsibilities
Design and execute test plans for AI/ML-driven features, including model outputs, prompts, and integrated application behavior
Build and maintain automated test suites covering functional, regression, integration, and API testing
Evaluate model outputs for accuracy, consistency, bias, hallucination, and edge-case failures
Develop evaluation frameworks and golden datasets/test cases to benchmark model performance over time
Test prompt engineering changes, model version upgrades, and fine-tuning outputs for regressions
Perform adversarial and red-team style testing to surface safety, security, and robustness issues
Validate data pipelines feeding into AI models, including data quality, schema, and drift detection
Collaborate with data scientists and ML engineers to define acceptance criteria and quality metrics for models
Test latency, scalability, and reliability of AI services under load
Contribute to CI/CD pipelines, integrating automated and model-evaluation tests
Design and execute test cases for LLM-based products such as chatbots, copilots, RAG systems, and AI agents, with focus on non-deterministic and generative outputs
Show the full description
Build evaluation suites using AI/LLM evaluation frameworks, including scoring rubrics and golden datasets
Validate prompt changes and model/version upgrades against baseline evaluation sets
Write evaluation scripts, test harnesses, and data validation logic in Python
Qualifications
Required
5+ years of quality assurance or testing experience
Hands-on experience testing LLM/generative AI products
Expertise in Python for test automation and evaluation scripting
Proficiency in SQL and data validation techniques
Practical experience with AI/LLM evaluation frameworks such as Ragas, DeepEval, LangSmith, Promptfoo, OpenAI Evals, or TruLens
Working knowledge of evaluation metrics for generative AI, including hallucination rate, faithfulness/groundedness, relevance, answer correctness, toxicity/bias scoring, and semantic similarity measures
Experience with prompt regression testing and validating prompt changes against baseline evaluation sets
Test automation and automated test suite development
Preferred
Prompt engineering or prompt testing experience
CI/CD pipeline automation experience
Exposure to Azure, Snowflake, or cloud data platforms