Job Overview
Penguin Computing is seeking an experienced AI QA / SDET Engineer to join our Software Engineering team. Penguin Computing’s Scyld Software products are used to deploy, provision, manage, and monitor some of the largest computational systems in the world. As we expand these platforms with AI/LLM-powered and agentic capabilities, this role will drive quality engineering across both our core infrastructure software and emerging AI solutions.
You will join our agile Software team and collaborate closely with Software Engineers, AI Engineers, Product Owners, Product Managers, and other teams across the organization to ensure our software meets the highest standards of quality, reliability, performance, and security.
The ideal candidate combines a strong foundation in software testing and automation with hands-on knowledge of LLM evaluation, AI agent testing, AI security testing, and modern AI quality methodologies. You will help define testing strategies, build scalable automation and AI evaluation frameworks, and continuously improve our testing processes and tools.
This role requires strong technical problem-solving skills, initiative, and the ability to work with a high degree of independence. You will have the opportunity to shape our evolving AI quality engineering practice and ensure that both traditional software and AI-powered capabilities are reliable, secure, measurable, and production-ready.
Roles & Responsibilities
Define and execute end-to-end QA strategy for software, AI/LLM features, and AI agents.
Design and maintain scalable test automation frameworks covering API, UI, integration, regression, performance, and infrastructure testing.
Build LLM/AI evaluation frameworks and automated evaluation pipelines integrated with CI/CD.
Define evaluation datasets, test scenarios, golden datasets, and quality benchmarks for AI features.
Evaluate LLM outputs for correctness, relevance, groundedness, hallucination, consistency, and task completion.
Implement automated evaluation methodologies including LLM-as-a-Judge, deterministic checks, model-based grading, and human evaluation.
Test AI agents and agentic workflows including reasoning, tool/function calling, multi-step execution, state management, error handling, and recovery.
Validate RAG pipelines, including retrieval quality, context relevance, grounding, citation accuracy, and response quality.
Perform adversarial and security testing of AI systems, including prompt injection, jailbreaks, data leakage, unsafe tool execution, excessive agency, and access-control violations.
Apply AI security methodologies such as OWASP Top 10 for LLM Applications and use appropriate AI red-teaming/security testing tools.
Build automated regression suites to identify behavioral and quality degradation across prompt, model, agent, and application changes.
Define AI quality metrics, dashboards, release gates, and acceptance criteria.
Test AI systems across different models and configurations to validate reliability and model-independent behavior.
Replicate customer environments through simulated data and usage patterns for realistic end-to-end testing.
Partner with Software Engineering, AI Engineering, Product, and Architecture teams to embed quality engineering early in the SDLC.
Troubleshoot complex issues across Linux, Kubernetes, networking, infrastructure, and AI application layers.
AI QA Methodologies & Tools
Experience with one or more of the following is highly desirable
LLM Evaluation: LLM-as-a-Judge, golden datasets, pairwise evaluation, rubric-based scoring, human-in-the-loop evaluation
Evaluation Frameworks: DeepEval, Ragas, Promptfoo, LangSmith, OpenAI Evals or equivalent
Agent Testing: tool/function-call validation, trajectory evaluation, task-completion testing, multi-agent workflow testing
RAG Evaluation: retrieval precision/recall, context relevance, faithfulness/groundedness, answer correctness
AI Security: prompt injection, jailbreak testing, data-exfiltration testing, privilege/access-control testing, adversarial testing
Security Frameworks/Tools: OWASP LLM guidance, Garak, PyRIT, Promptfoo or equivalent AI red-teaming tools
Observability: tracing and evaluation of prompts, model responses, tool calls, agent decisions, latency, token usage, and failures
Qualifications
Degree in Computer Science or related field, or equivalent professional experience.
6-8 years of experience in QA, SDET, software engineering, or test automation.
Strong understanding of QA methodologies, test architecture, and software development lifecycle.
Strong coding/scripting skills in Python; Bash, JavaScript, or C experience is beneficial.
Experience building automation frameworks and integrating automated tests into CI/CD pipelines.
Strong Linux command-line, debugging, and troubleshooting skills.
Experience with API, functional, integration, regression, performance, and security testing.
Experience with Kubernetes, containers, and virtualization.
Hands-on exposure to LLMs, Generative AI applications, RAG, AI agents or Agentic AI Platforms.
Understanding of LLM-specific failure modes including hallucination, non-determinism, prompt injection, and unsafe agent behavior.
Experience with Selenium, Playwright, PyTest, or similar testing frameworks.
Knowledge of HPC/AI infrastructure, networking, GPUs, or distributed systems is a plus.
What Success Looks Like
Build a modern quality engineering practice where traditional software testing and AI evaluation come together, enabling us to ship infrastructure software and AI agents that are reliable, measurable, secure, and safe to operate in production.
Location
Bangalore
Travel
Apply now »
Find similar jobs
CAREER OVERVIEW
VIEW ALL JOBS
TOP JOB SEARCHES
CONNECT WITH US
Opens in a new tab.
Opens in a new tab.
Opens in a new tab.
Opens in a new tab.
Copyright © 2025 Penguin Solutions®. All Rights Reserved.
