Build automated validation pipelines for LLM outputs using structured assertions, semantic scoring, embeddings, and evaluation metrics.
Create synthetic datasets and benchmark datasets for AI testing scenarios.
Test multi-agent orchestration workflows and agent memory/context retention.
Validate MCP (Model Context Protocol) integrations and tool interoperability.
Work on Context Engineering, Prompt Engineering, and Prompt Optimization strategies.
Evaluate latency, token usage, model cost optimization, throughput, and scalability.
2. Automation Engineering
Develop and maintain scalable UI/API automation frameworks using: Python (mandatory)Robot Framework (preferred)Playwright/Selenium/Cypress.
Automate testing for: REST APIs GraphQL APIs Event-driven systems Streaming systems Webhooks Microservices MFE (Micro Frontend) architectures.
Build reusable automation libraries, utilities, and AI-assisted testing workflows.
Integrate automation into CI/CD pipelines and quality gates.
3. AI Engineering & Platform Validation
Validate RAG architectures including Chunking strategies Embedding quality Retrieval accuracy Re-ranking Vector search relevance
Work with vector databases such as: Pinecone Weaviate ChromaDB Qdrant FAISS
Build or validate AI agents leveraging: LangChain, LangGraph, CrewAI, AutoGen, Semantic Kernel OpenAI SDKs
Validate AI workflows involving: Tool calling Function calling Agent memory Agent planning/execution Multi-modal AI systems Structured outputs
4. DevOps, Observability & Reliability
Build and maintain quality workflows integrated with: GitHub Actions Jenkins GitLab CI/CD Azure DevOps
Work with Docker and Kubernetes environments.
Perform cloud-based testing and deployments across AWS/Azure/OCI/GCP.
Validate distributed systems reliability and resiliency.
Monitor AI system behaviors using observability platforms/logging/tracing tools.
Contribute to shift-left and quality engineering best practices.
Who you are & what you'll need to succeed.
Bachelor’s/Master’s degree in Computer Science, Engineering, AI/ML, or related field preferred.
Equivalent hands-on experience and demonstrated projects are highly valued.
Strong programming experience in Python
Strong understanding of Software Testing & Quality Engineering principles
Hands-on experience with: AI/LLM testing Prompt Engineering Context Engineering AI Evals RAG systems AI Agents
Experience with: CI/CD pipelines Docker & KubernetesCloud platforms (AWS/Azure/OCI/GCP) API testing & automation Microservices testing UI automation
Experience with SQL and data validation
Strong debugging, analytical, and troubleshooting skills
Experience working in Agile/Scrum environments
Good to have skills
Experience with Robot Framework
Experience with Playwright
Knowledge of: LangChain, LangGraph, LlamaIndex CrewAI Semantic Kernel MCP (Model Context Protocol)
Hands-on experience with: Cursor GitHub Copilot Claude Gemini OpenAI APIs
Experience with vector databases and embeddings
Exposure to MLOps workflows and ML lifecycle testing
Knowledge of performance testing and security testing
Experience validating AI governance/compliance requirements
Experience with contract testing, chaos testing, or resilience engineering
Familiarity with accessibility testing and cross-browser/device testing
Exposure to synthetic monitoring and production observability
Bonus Skills (High Impact)
AI Red Teaming
LLM Security Testing (Prompt Injection, Jailbreaks, Data Leakage)
Agent Reliability & Safety Testing
AI Cost Optimization
Distributed Systems Testing
Test Data Engineering
Feature Flag Testing
AI Observability & Telemetry
Reinforcement Learning feedback workflows
Human-in-the-loop evaluation systems
Knowledge Graph integrations
Multi-modal AI testing (vision/audio/text)
What We’re Looking For
Engineers who actively experiment with AI tools and build side projects, prototypes, PoCs, or open-source contributions.
Strong problem solvers who can combine traditional QE practices with modern AI-native engineering approaches.
Passion for innovation, developer productivity, automation acceleration, and intelligent testing.
Ability to rapidly learn emerging AI frameworks, tools, and architectures.
Nice Portfolio Signals
Candidates with one or more of the following will stand out
GitHub repositories showcasing AI/automation projects
AI agents or copilots built personally or professionally
Contributions to open-source automation or AI ecosystems
Blogs, talks, workshops, or technical content on AI/testing
Experience integrating AI into SDLC or QE workflows
Experience with autonomous testing systems or self-healing automation
Why you'll love us.
Work-life balance. Our work is thrilling and meaningful, but we know balance is key to living well.
We celebrate one another’s differences. We’re proud of our culture of diversity and inclusion. We foster a culture of belonging. Our company and customers benefit when employees bring their authentic selves to work. We have programs that bring us together on important issues and provide learning and development opportunities for all employees. We have 20 + affinity groups where you can network and connect with Newfolders globally.
We care about you. We are a family, and we care about you and your family’s physical and mental health by providing competitive HMO benefits – 175k MBL with one free dependent upon one year of service! We also give out Punctuality Bonus, Generous Vacation policy, and much more!
Where can we take you? We’re fans of helping our employees learn different aspects of the business, be challenged with new tasks, be mentored, and grow their careers. Unfold new possibilities with #teamnewfold!
This Job Description includes the essential job functions required to perform the job described above, as well as additional duties and responsibilities. This Job Description is not an exhaustive list of all functions that the employee performing this job may be required to perform. The Company reserves the right to revise the Job Description at any time, and to require the employee to perform functions in addition to those listed above.
