Key Responsibilities
Design and document comprehensive evaluation frameworks for Aircall’s AI agents across voice, chat and messaging.
Train and fine-tune voice models (TTS, ASR, speech-to-speech) using production and synthetic data, iterating on architecture, data mix, and training strategy to improve accuracy, naturalness, and latency.
Assess AI generated solutions across training pipelines, experimentation setups, debugging processes, and optimization strategies.
Analyze system design decisions and identify strengths, weaknesses, and potential failure points.
Design annotation guidelines and workflows for human-labeled evaluation data, and calibrate LLM-as-judge systems against human raters to ensure automated evals stay trustworthy over time.
Build and maintain live quality monitoring for deployed AI agents, tracking accuracy, resolution rate, and safety signals in production, and flagging model or data drift before it impacts customers.
Own the metric contract for every published AI metrics, including definition, population, grain, rollup, validity window.
Build release gates, the offline regression suite each AI surface must pass before a prompt, model, or config change ships, measuring reliability across repeated trials, not just average pass rates.
Build voice-specific evaluation: simulated callers across accents, languages, background noise, barge-in, DTMF, and tool failures, with latency and ASR accuracy as first-class quality metrics.
Minimum Qualifications
BS in Computer Science, Machine Learning, Statistics, or related field
3+ years of experience in ML Engineering or Applied ML with 8+ years of overall experience
Strong experience in evaluating supervised, unsupervised, LLMs and deep learning models.
Hands-on experience in failure analysis and evaluating LLMs
Experience building automated evaluation systems
Strong communication skills to articulate complex technical concepts across technical and non-technical audiences
Hands-on experience training or fine-tuning voice/speech models (TTS, ASR, or speech-to-speech), including data pipeline construction and experimentation.
Preferred Qualifications
MS / PhD in Computer Science, Machine Learning, Statistics, or related field
Experience evaluating LLMs or agentic systems (e.g., LLM-as-a-judge, RAG evaluation)
Experience with synthetic data generation and prompt engineering
Experience training or fine-tuning voice models at scale, with familiarity in synthetic data generation, model distillation, or low-latency inference optimization for production voice agents.
Base salary range
$181,000—$250,000 USD
Why join us?
🚀 Key moment to join Aircall in terms of growth and opportunities
💆♀️ Our people matter, work-life balance is important at Aircall
📚 Fast-learning environment, entrepreneurial and strong team spirit
🌍 45+ Nationalities: cosmopolite & multi-cultural mindset
💶 Competitive salary package & benefits
DE&I Statement
At Aircall, we believe diversity, equity and inclusion – irrespective of origins, identity, background and orientations – are core to our journey.
We pride ourselves on promoting active inclusion within our business to foster a strong sense of belonging for all. We’re working to create a place filled with diverse people who can enrich and learn from one another. We’re committed to ensuring that everyone not only has a seat at the table but is valued and respected at it by providing equal opportunities to develop and thrive.
We will constantly challenge ourselves to make sure that we live up to our ambitions around diversity, equity and inclusion, and keep this conversation open. Above all else, we understand and acknowledge that we have work to do and much to learn.
Want to know more about candidate privacy? Find our Candidate Privacy Notice here.
We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analyzing resumes, or assessing responses. These tools assist our recruitment team but do not replace human judgment. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed, please contact us.
