Create and maintain representative test datasets, expected outcomes, and evaluation rubrics. Use deterministic assertions, LLM-as-judge, and human review where each is appropriate
Run quality checks for every release and for material prompt, model, tool, or data-source change. Integrate repeatable tests into CI/CD pipelines as release gates
Monitor production quality using measures such as task completion, answer accuracy, groundedness, hallucinations, tool-call success, latency, cost, and user feedback
Detect drift and quality regressions, set alert thresholds, investigate failures, and expand test coverage based on production findings
Evaluate risks including prompt injection, sensitive-data exposure, bias, and unauthorized or unsafe agent actions
Publish quality dashboards and release-readiness reports. Partner with engineering, product, and SRE teams on defect triage, root-cause analysis, and corrective actions
Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements
Required Qualifications
Bachelor's degree in Computer Science or a related field, or equivalent practical experience
5+ years of experience in software quality engineering or test automation, including 2+ years testing AI/ML, generative AI, or LLM-based systems
Hands-on experience evaluating LLM applications with test datasets, rubrics, groundedness or relevance checks, and a combination of automated and human review
Experience designing and executing manual and automated functional, API, integration, end-to-end, regression, and risk-based tests
Experience integrating automated quality checks into CI/CD pipelines and testing APIs, databases, and distributed applications
Experience using logs, traces, dashboards, alerts, and user feedback to investigate production quality issues
Solid Python skills and experience building or extending test automation frameworks, such as Pytest or Playwright
Working knowledge of QA methods, defect lifecycle management, Agile delivery, and the software development lifecycle
Understanding of RAG, prompt engineering, tool use, and multi-step agent workflows
Demonstrated solid analytical, problem-solving, communication, and collaboration skills
Preferred Qualifications
Experience with evaluation or observability tools such as LangSmith, Langfuse, Arize, DeepEval, Ragas, Vertex AI evaluation, or OpenTelemetry
Experience with GCP, Vertex AI, or Gemini; experience testing applications on AWS or Azure
Experience with JavaScript/TypeScript or Java, Selenium, advanced Playwright framework development, or browser testing platforms such as BrowserStack or Sauce Labs
Experience with adversarial testing, responsible AI, and the OWASP Top 10 for LLM Applications
Experience with A/B testing or canary evaluation of prompt and model changes
Experience testing conversational AI, multi-agent or MCP-enabled solutions, event-driven services, or API orchestration
Experience using AI to assist with test generation, failure analysis, or quality reporting
Experience with performance and load testing tools such as JMeter or LoadRunner, or operational tools such as Dynatrace or SonarQube
Experience with cloud migration, data migration, or data reconciliation testing
Healthcare or other regulated-industry experience, including knowledge of HIPAA, data privacy, and security requirements. Experience with claims, eligibility, prior authorization, provider, contact center, or digital health systems
At UnitedHealth Group, our mission is to help people live healthier lives and make the health system work better for everyone. We believe everyone-of every race, gender, sexuality, age, location and income-deserves the opportunity to live their healthiest life. Today, however, there are still far too many barriers to good health which are disproportionately experienced by people of color, historically marginalized groups and those with lower incomes. We are committed to mitigating our impact on the environment and enabling and delivering equitable care that addresses health disparities and improves health outcomes - an enterprise priority reflected in our mission.