Utilize RAGAS or similar frameworks to evaluate LLM outputs and identify opportunities for optimization.
Conduct end-to-end testing from data ingestion to user-facing output.
Validate system stability across GCP cloud components, including compute, storage, networking, and containers.
Collaborate with DevOps and ML engineers to triage bugs, improve model performance, and ensure system reliability.
Ensure test coverage spans multiple deployment environments, including Kubernetes clusters and cloud VMs.
KEY SUCCESS FACTORS
3+ years of QA Automation Engineering experience
Experience with Playwright for automated application testing, including sign-in and SSO authentication flows
Ability to design tests that validate output accuracy and system behavior across different user flows
Experience with Python API testing using requests, Pytest, and integration into CI/CD and cloud environments.
Experience with ML/LLM Evaluation
Hands-on experience implementing LLM-as-a-Judge evaluation methodologies using models such as Gemini, Claude, or similar LLMs
Experience designing evaluation criteria and assessing LLM outputs for accuracy, relevance, factuality, and quality
Understanding of human evaluation and data annotation processes, including leveraging feedback loops to improve and fine-tune model performance
Experience with Cloud Platforms (GCP preferred)
Understanding of modern cloud architecture and how distributed components interconnect in production environments
NICE TO HAVES
Experience with multilingual LLM evaluation, localization testing, and human-in-the-loop evaluation workflows
About INSPYR Solutions
Technology is our focus and quality is our commitment. As a national expert in delivering flexible technology and talent solutions, we strategically align industry and technical expertise with our clients' business objectives and cultural needs. Our solutions are tailored to each client and include a wide variety of professional services, project, and talent solutions. By always striving for excellence and focusing on the human aspect of our business, we work seamlessly with our talent and clients to match the right solutions to the right opportunities. Learn more about us at inspyrsolutions.com. INSPYR Solutions provides Equal Employment Opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, sex, national origin, age, disability, genetics, or any other protected status. INSPYR Solutions complies with all applicable laws governing nondiscrimination in employment in every location in which the company has facilities. Applicants requiring reasonable accommodation during the application or interview process should contact HR@inspyrsolutions.com for assistance.
Information collected and processed through your application with INSPYR Solutions (including any job applications you choose to submit) is subject to INSPYR Solutions' Privacy Policy and INSPYR Solutions' AI and Automated Employment Decision Tool Policy: https://www.inspyrsolutions.com/policies/. By submitting an application, you are consenting to being contacted by INSPYR Solutions through phone, email, or text.
26-160430