Develop services and pipelines for collecting and processing agent traces, tool calls, workflow outcomes, user feedback, and other production signals.
Build systems that evaluate and verify model behavior at scale using deterministic checks, model-based evaluation, statistical analysis, and human review where appropriate.
Create reliable feedback loops between production systems and model-training teams, turning observed failures and successful behaviors into evaluation cases, regression tests, and candidate training data.
Build training and evaluation environments that represent realistic business workflows through stable APIs, tools, simulations, resettable scenarios, and reproducible state.
Implement the integrations and supporting services required for agents to interact with enterprise systems, domain data, and multi-step workflows.
Design for reproducibility across models, prompts, tools, datasets, environments, and application versions so that changes can be measured with confidence.
Partner with model-training teams to integrate new models, investigate behavior failures, define engineering requirements, and determine whether problems are best addressed through software, tools, data, evaluation, or model changes.
Improve reliability and operability through observability, automated testing, failure recovery, safe rollout mechanisms, and clear launch criteria.
Set a high engineering bar through system design, code review, documentation, mentoring, and pragmatic technical leadership across teams.
What We're Looking For
Around eight or more years of software engineering experience, or equivalent depth, with a track record of designing and delivering complex production systems.
Strong Python and backend engineering expertise, including API design, distributed systems, data processing, automated testing, and maintainable service architecture.
Hands-on experience building LLM-powered applications, AI agents, machine learning platforms, or similarly complex systems.
Practical understanding of AI evaluation, including test-case design, trace analysis, failure classification, regression testing, and measurement of nondeterministic systems.
Strong understanding of software engineering fundamentals, including code quality, security, observability, CI/CD, and operational reliability.
Experience with cloud platforms such as Azure, AWS, or GCP, along with containerization and modern deployment practices.
The ability to work effectively with researchers and model-training engineers, translating experimental needs into well-designed software and production evidence into actionable feedback.
Strong technical judgment, clear communication, and the ability to provide direction in ambiguous problem spaces without becoming a bottleneck for delivery.
Preferred Qualifications
Familiarity with model-training or post-training approaches such as supervised fine-tuning, preference optimization, reinforcement learning, reward or verifier design, and checkpoint evaluation.
Experience building model-training environments, simulators, evaluation harnesses, or agent development platforms.
Understanding of training and evaluation data practices, including curation, synthetic data, provenance, versioning, quality control, and prevention of data leakage.
Experience with supply chain, retail, or other enterprise environments involving complex workflows, permissions, business constraints, and human approval paths.
What Makes This Role Different
This role sits between model development and production engineering. You will not be asked to conduct research in isolation or simply wrap a model in an application. You will build the systems that make model behavior measurable, reproducible, improvable, and useful in real operational environments.
Our Values
If you want to know the heart of a company, take a look at their values. Ours unite us. They are what drive our success – and the success of our customers. Does your heart beat like ours? Find out here: Core Values
All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability or protected veteran status.
