What You Will Do
Own the evaluation platform component from architecture and delivery through adoption, operations, and continuous improvement; define its roadmap, service boundaries, integration contracts, and engineering standards.
Build self-service APIs, SDKs, developer integrations, and user interfaces for configuring evaluations, submitting and tracking runs, and accessing results within existing engineering workflows.
Create an extensible integration layer for evaluation frameworks, scorers, model endpoints, and agent runtimes, enabling teams to use supported tools and add evaluators through consistent interfaces.
Engineer reliable evaluation execution services with asynchronous orchestration, scheduling, concurrency management, retries, cancellation, and workload isolation across projects and teams.
Provide versioning and lifecycle management for datasets, evaluation configurations, evaluator definitions, and result artifacts, with metadata and lineage that make runs reproducible and comparable.
Support agentic evaluation through common contracts for inputs, outputs, traces, tool calls, and execution trajectories; provide controlled execution and replay patterns where appropriate.
Build reusable evaluator components and configuration patterns for output quality, retrieval, tool use, task completion, safety, latency, and cost, while enabling application teams to supply domain-specific evaluation content.
Integrate evaluation execution and results with CI/CD, experiment tracking, observability, and promotion workflows; enable configurable quality thresholds and evidence collection for release and governance review.
Deliver consistent results schemas, comparison views, scorecards, and reporting APIs so teams can examine trends and compare model, prompt, and agent versions within project context.
Embed authentication, authorization, project isolation, auditability, and data retention controls, working with security and governance partners to protect evaluation datasets, traces, and results.
Operate the component as a production service, defining service objectives, monitoring reliability and execution performance, managing capacity and cost visibility, and leading incident resolution and upgrades.
Drive adoption through documentation, onboarding patterns, reference integrations, and technical support; use platform telemetry and feedback to prioritize improvements and reduce integration effort.
What We Expect of You
Own evaluations as an enduring platform capability, with clear interfaces, dependable operations, and a roadmap informed by multiple teams.
Turn recurring application requirements into reusable capabilities and extension points, balancing standardization with flexibility across agent frameworks and workloads.
Measure success through platform adoption, service reliability, reproducibility, evaluation turnaround time, and developer experience.
Provide hands-on technical leadership through architecture and code reviews, mentoring, and collaboration across engineering and governance teams.
Basic Qualifications
Bachelor's or master's degree in computer science, AI, or a related field
10+ years of experience in software engineering, AI engineering, ML systems, or platform engineering
Strong Python and software engineering skills, including the design of production APIs, reusable libraries, and distributed backend services
Experience building and operating shared platform services or infrastructure consumed by multiple applications or engineering teams
Hands-on experience integrating AI evaluation frameworks, experiment tracking, or automated AI quality workflows into production engineering environments
Understanding of LLM and agentic architectures, including orchestration, RAG, tool use, state management, and the evaluation data and traces these systems produce
Experience with asynchronous processing, cloud services, CI/CD, observability, versioning, and secure access to enterprise systems
Ability to define service contracts, make architecture tradeoffs, and lead engineering delivery and operations across teams
Preferred Qualifications
Experience owning an evaluation platform component, shared testing infrastructure, or an ML platform service through design, adoption, and production operations
Experience integrating tools such as MLflow, RAGAS, DeepEval, or Promptfoo through reusable adapters and common configuration and results interfaces
Experience designing plugin architectures, workload isolation, job orchestration, artifact storage, and versioned APIs or schemas
Experience connecting agent traces, production feedback, and human review systems to shared evaluation infrastructure
Familiarity with the strengths and limitations of deterministic checks, LLM-as-judge scoring, and human assessment, and their implications for platform design
Experience with enterprise security and governance controls, usage metering, and capacity and cost management for AI services
Soft Skills
Strong technical leadership and mentoring, with ownership from architecture through production support
Ability to translate needs across application teams into a coherent platform roadmap
Clear communication of service capabilities, integration patterns, operating standards, and technical tradeoffs
Practical engineering judgment and a strong focus on developer experience and service reliability
Equal Opportunity Statement
Amgen is an Equal Opportunity employer and will consider you without regard to your race, color, religion, sex, sexual orientation, gender identity, national origin, protected veteran status, or disability status.
.