Selecting appropriate manual, deterministic, model-based and statistical evaluation methods.
Analysing errors by category, user journey, risk level and likely root cause.
Distinguishing model failures from failures in retrieval, chunking, ranking, prompts, tools, guardrails, integrations, source data and evaluators.
Validating automated evaluators, including LLM-as-a-Judge, against human judgement.
Detecting blind spots, noise, bias and misleading aggregation in evaluation results.
Creating baselines, controlled comparisons and regression tests that allow meaningful comparison between system versions.
Prioritising tests according to the probability and impact of failure rather than coverage alone.
Communicating results, uncertainty, limitations and residual risks in a form that supports release and product decisions.
Proactively identifying emerging failure modes based on system changes, production observations and research.
What we're looking for
Professional experience in quality engineering, software testing, test automation, data science, software engineering or a related field (we can consider different seniority levels).
Experience or strong interest in working with AI-powered solutions, either through developing, testing, evaluating or supporting them.
Good Python proficiency and experience writing clean, maintainable code for analysis, automation or testing purposes.
Interest in understanding how AI systems behave, how reliable they are, and how their quality can be measured.
Strong analytical and problem-solving skills, with the ability to investigate issues and identify root causes.
Ability to translate business objectives, customer outcomes and risks into practical evaluation approaches.
Strong communication skills and the ability to explain findings to both technical and non-technical stakeholders.
A curious, proactive mindset and a genuine interest in AI, emerging technologies and system robustness.
Nice to have
Experience evaluating, testing or monitoring AI-powered systems in production environments.
Working with structured data, especially JSON, logs and tabular evaluation results.
Understanding of LLM behaviour, tokenisation, context limits, parameters and non-determinism.
Understanding of multi-turn context, RAG, grounding, faithfulness, hallucinations, guardrails, attacks (like prompt injection etc.) and tool/function calling.
Software engineering practices: Git, modular code, configuration management and automated tests.
In Aviva Services Excellence Centre, we provide
Contract of employment
Performance Bonus
Private medical care (ENEL-MED)
Cafeteria system of benefits (e.g. MultiSport card)
Additional parent privileges
baby leave – 6 weeks leave for second parent
shortened working time after parental leave
Employer-funded group insurance
Employee Shares Plan
Extra day off for birthday
Paid time off to volunteer
Additional leave acquired over the years in the company (even 5 days more in a year)
Wellbeing Leave - additional two weeks paid leave after 10 years of service
Psychological support consultations
Access to the e-learning platforms
Internal employee referral program
Hybrid working model from Poland with a few days in Warsaw office per month
Attractive office location in Warsaw - Gdański Business Center
Additional entitlements for people with disabilities - Work Without Barriers
Work environment that promotes diversity - Diversity Charter
Do you see yourself in Aviva Services Excellence Centre? Be sure to apply!
Aviva is for everyone!
We’ve been ranked the #1 Best Workplace in Poland for a second year in a row and a top choice for Millennials by Great Place to Work®!
As a signatory to the Diversity Charter, Aviva applies a policy of equal treatment and creates a work environment that is friendly to people with disabilities. We do not require from candidates the date of birth, marital status and attached photos to your CV. We encourage all individuals regardless of gender to apply.
Most of our people are smart working – spending also time in our office - combining the benefits of flexibility, with time together with colleagues.
We’re inclusive - we welcome applications from people with diverse backgrounds and experiences.
Excited but not sure you tick every box? Research tells us that women, particularly, feel this way. So, regardless of gender, why not apply and try!
To find out more about current job offers at Aviva take a look here.
