Combining test methods and validating automated evaluators against human judgement.
Using baselines, key metrics and risk-based regression tests to support release decisions.
Integrating evaluation into CI/CD, observability and engineering workflows with traceability across datasets, prompts, models and metrics.
Producing concise, auditable findings that explain confidence, uncertainty, customer impact and residual risk.
Partnering with product, engineering, architecture, data, risk, compliance and quality stakeholders to influence delivery and release decisions.
Coaching and guiding AI Quality Engineers, planning team work, reviewing outputs and building a collaborative culture of constructive challenge and learning.
What we're looking for
Several years of relevant professional experience in quality engineering, software testing, software engineering, data science or a related technical field.
Hands-on experience evaluating AI, large language model, RAG or agent-based applications.
Practical Python skills and experience with APIs, logs, traces, structured datasets and SQL or similar
analysis.
Ability to design repeatable evaluations and automation, not just manual testing or vendor scoring.
Good understanding of sampling, thresholds, trends and non-deterministic system behaviour.
Experience with automated testing, version control, CI/CD and maintainable engineering practices.
Ability to turn technical findings into decision-quality evidence for business, risk and engineering stakeholders.
Experience coaching or leading engineers in an emerging or ambiguous domain.
Strong communication and influencing skills, with the confidence to challenge weak evidence.
Nice to have
Experience leading projects, coordinating technical workstreams or managing people.
Experience with AI evaluation or observability tools such as DeepEval, Arize or similar platforms.
Experience designing shared frameworks, reusable services or platform capabilities.
Knowledge of responsible AI, model risk, controls, privacy or customer-outcome requirements.
Experience in financial services, insurance or another regulated industry.
Exposure to cloud AI systems, observability and production support.
In Aviva Services Excellence Centre, we provide
Contract of employment
Performance Bonus
Private medical care (ENEL-MED)
Cafeteria system of benefits (e.g. MultiSport card)
Additional parent privileges
baby leave – 6 weeks leave for second parent
shortened working time after parental leave
Employer-funded group insurance
Employee Shares Plan
Extra day off for birthday
Paid time off to volunteer
Additional leave acquired over the years in the company (even 5 days more in a year)
Wellbeing Leave - additional two weeks paid leave after 10 years of service
Psychological support consultations
Access to the e-learning platforms
Internal employee referral program
Hybrid working model from Poland with a few days in Warsaw office per month
Attractive office location in Warsaw - Gdański Business Center
Additional entitlements for people with disabilities - Work Without Barriers
Work environment that promotes diversity - Diversity Charter
Do you see yourself in Aviva Services Excellence Centre? Be sure to apply!
Aviva is for everyone!
We’ve been ranked the #1 Best Workplace in Poland for a second year in a row and a top choice for Millennials by Great Place to Work®!
As a signatory to the Diversity Charter, Aviva applies a policy of equal treatment and creates a work environment that is friendly to people with disabilities. We do not require from candidates the date of birth, marital status and attached photos to your CV. We encourage all individuals regardless of gender to apply.
Most of our people are smart working – spending also time in our office - combining the benefits of flexibility, with time together with colleagues.
We’re inclusive - we welcome applications from people with diverse backgrounds and experiences.
Excited but not sure you tick every box? Research tells us that women, particularly, feel this way. So, regardless of gender, why not apply and try!
To find out more about current job offers at Aviva take a look here.