Collaboration: Work closely with AI Engineers to translate "vague" business requirements into measurable, automated test cases.
Required Skills & Qualifications
Experience: 10+ years in QA Automation, with a recent focus on AI/ML or LLM-based applications.
Python Proficiency: Expert-level Python skills (the industry standard for AI testing) and experience with testing frameworks like Pytest.
AI Testing Tools: Familiarity with AI evaluation frameworks such as LangSmith, DeepEval, RAGAS, or Promptfoo.
API & Backend Testing: Deep experience with Playwright, Selenium, or Cypress for UI, but a heavy focus on API-level testing and database validation.
Statistical Mindset: Understanding that AI testing often requires "scoring" (e.g., 85% accuracy) rather than a simple binary pass/fail.
Data Skills: Ability to work with SQL and JSON to validate data retrieved by agents during RAG (Retrieval-Augmented Generation) processes.
Preferred Qualifications
Experience testing Multi-Agent Systems (where one agent tests another).
Knowledge of Prompt Engineering and how it influences software behavior.
Background in Investment Banking or Fintech (if applicable) to understand high-stakes data accuracy.
Compensation, Benefits and Duration
Minimum Compensation: USD 38,000
Maximum Compensation: USD 133,000
Compensation is based on actual experience and qualifications of the candidate. The above is a reasonable and a good faith estimate for the role.
Medical, vision, and dental benefits, 401k retirement plan, variable pay/incentives, paid time off, and paid holidays are available for full time employees.
This position is not available for independent contractors
No applications will be considered if received more than 120 days after the date of this post