AI Test Engineer
The role in one line: A hands-on engineer who proves our AI features are accurate, safe and stable before and after every change, and who can build that testing setup alone.
What this person will actually do
Build automated accuracy checks for the AI
Make those checks a gate in the CI pipeline,]
Build and maintain test datasets
Test the whole pipeline end to end
Watch quality in production using LLM tracing tools (Arize Phoenix, LangSmith) to spot drift, rising failures and cost or latency jumps.
Run safety and robustness testing: prompt injection inside documents, hallucinated fields, PHI leaking into logs, and inconsistent answers to the same input.
Produce evidence for a regulated product. Test reports need to stand up in a medical-device-style audit (IEC 62304, HIPAA).
Must-have skills (screen hard on these)
Strong Python test automation: pytest, API testing, CI pipelines (GitLab).
LLM evaluation hands-on: has built eval sets and scoring, not just “used ChatGPT.” Ask them to name the metrics they used and how they set pass/fail thresholds.
Experience with at least one eval or tracing tool: LangSmith, Arize Phoenix, DeepEval, Ragas, promptfoo, or similar.
Understands non-deterministic testing: how to test something that gives slightly different answers each run (repeat runs, tolerances, LLM-as-judge and its limits).
Document or vision AI exposure: OCR or extraction from PDFs and images is a big plus for us.
Show the full description
Solo-capable: has set up a test approach from scratch, not only executed someone else’s.
Good-to-have
LangChain / LangGraph. Useful, but not essential: our AI Doc calls Gemini directly through Google’s SDK. Don’t reject strong candidates for lacking LangGraph.
Google Cloud / Vertex AI, Django APIs, Playwright.
Healthcare or regulated-domain testing, and handling of PHI and de-identified data.
Performance and cost testing of LLM calls (latency, tokens).