Create evaluation frameworks (offline test sets, LLM-as-judge, regression suites, human review loops) to measure accuracy, hallucination, latency and cost.
Integrate agents with CRMs, ERPs, ticketing systems and third-party APIs so they can complete tasks end to end.
Build guardrails, escalation logic and human-in-the-loop handoffs for sensitive or low-confidence situations.
Monitor production agents, analyze conversation failures, and continuously improve prompts, models and retrieval.
Evaluate and select foundation models, and apply fine-tuning or distillation where it improves quality or cost.
Work with Product, Engineering, CX Operations and Solutions teams to translate customer requirements into scalable AI capabilities.
Stay current on LLM, agentic and speech/voice AI developments, and bring practical improvements into the platform.
This is a Bangalore-based role.
What does success look like in this role?
Your agents resolve customer queries accurately and safely at scale, with measurable gains in automation rate, resolution quality and cost per conversation. Issues are caught by your evaluation and monitoring before customers notice them, and new AI capabilities move from prototype to production quickly and reliably.
What would make you a good fit for this role?
2–5 years of software engineering experience, with at least 1 year building LLM/Gen AI applications in production.
Strong Python skills and experience building backend services and APIs.
Hands-on experience with LLM APIs (OpenAI, Anthropic, Gemini or open-source models) and frameworks or patterns for agents, tool calling and RAG.
Working knowledge of vector databases, embeddings and search/retrieval techniques.
Experience with prompt engineering and with evaluating LLM output quality.
Exposure to conversational AI, chatbots, voice AI or automation platforms is a strong plus.
Familiarity with cloud platforms, containers and CI/CD.
Bachelor’s degree in Computer Science, Engineering or a related field.
What are the most critical skills for this role?
Building and debugging production-grade LLM agents and RAG systems
Designing evaluations and iterating on measurable quality
Structured problem solving and ownership from prototype to production
Clear communication with technical and business stakeholders
Comfort working in a fast-moving, customer-driven environment
Here’s what you will gain from this role
Hands-on exposure to large-scale CX automation and AI-powered platforms.
Real production scale: enterprise agents handling millions of customer interactions.
Direct impact on a product used by leading global brands.
Close collaboration with product, engineering and customer teams.
Learning and growth in a rapidly evolving agentic AI space.
