Evaluation is a core responsibility: the role leads the development and improvement of reusable evaluation capabilities for large language models (LLMs) and AI applications. It establishes meaningful benchmarks, helps teams understand performance and limitations, and uses the findings to guide model selection and improvements in quality, reliability, latency, and cost.
This is a hands-on individual-contributor role with no people-management responsibilities. Leadership is exercised through technical ownership, direct contribution, mentoring, and influence. Clear communication is essential to aligning teams, explaining technical trade-offs, and keeping stakeholders informed of decisions, progress, and risks.
How you will add value
Lead technical delivery: Translate business needs into practical designs, assessing feasibility, dependencies, and AI-assisted versus traditional approaches. Build critical components, review code, and guide initiatives through testing, deployment, and production readiness.
Own the evaluation framework: Develop reusable tools for LLM and application evaluation, including benchmarks, representative datasets, scoring methods, and regression tests. Combine automated checks, LLM-as-judge methods, and human review to assess accuracy, grounding, safety, and reliability.
Optimize quality and performance: Apply advanced prompting and improve retrieval, context construction, model selection, routing, and serving approaches. Use appropriate tuning, caching, and batching to improve quality and throughput while reducing latency and unit cost.
Validate improvements with evidence: Run structured experiments and A/B tests with clear hypotheses, guardrails, and rollback criteria. Compare models, techniques, and vendors against internal benchmarks, document limitations, and use findings to prioritize changes.
Align teams and communicate clearly: Partner across business and technical teams to resolve dependencies and meet enterprise standards. Explain trade-offs and evaluation results, document decisions, and communicate progress, risks, and next steps.
Scale effective practices: Mentor engineers and share reusable components, evaluation tooling, and practical guidance. Apply AI where it improves testing, code review, documentation, and monitoring, and measure the benefits.
Technical Environment
Core stack: Python, LLM and machine-learning frameworks, evaluation and experimentation libraries, and retrieval and vector tooling for AI applications.
Platforms: Cloud or hybrid model platforms, approved LLM endpoints, model gateways and registries, and GPU or accelerated compute where required by the solution.
Engineering workflow: Version-controlled code and experiments, automated tests, reproducible evaluations, continuous integration, and staged deployment.
Quality and reliability: Application and model monitoring, quality benchmarks, latency and cost metrics, regression detection, safety guardrails, and feedback from production use.
What will help you be successful in this role
Bachelor's degree in Computer Science, Engineering, or a quantitative discipline, or equivalent practical experience, and six or more years in applied machine learning or AI engineering, including production LLM or model-serving systems. An advanced degree is preferred.
Demonstrated technical leadership from ambiguous requirements through delivery, with the ability to align cross-functional partners, mentor engineers, and balance new initiatives with ongoing optimization without formal management authority.
Strong hands-on Python development and software-engineering judgement, including solution design, code review, testing, troubleshooting, and building maintainable production components.
Hands-on experience evaluating LLMs and AI applications using representative datasets, scoring methods, automated tests, and failure analysis. Familiarity with LLM-as-judge methods and human evaluation, including validating results against expert judgement and using findings to compare models and identify regressions.
Strong experience in prompt and retrieval optimization and model performance tuning, with practical evidence of improving quality, latency, reliability, and cost in production systems.
Rigorous experimental discipline and statistical judgement, including designing meaningful comparisons, accounting for variability in model outputs, interpreting results, and recognizing limitations.
Strong written and verbal communication: explain technical concepts and trade-offs to nonspecialists, present clear recommendations, document decisions, and communicate risks and dependencies early.
Nice to have skills
Experience in asset management, wealth management, or other financial-services AI applications.
Experience with fine-tuning, preference optimization, quantization, or distillation of production models.
Experience with GPU performance engineering or inference-server optimization.
Experience with LLM observability tooling, production drift detection, or evaluation practices in regulated environments.
Work Schedule & Locations
This position will work a hybrid schedule, in the office 3 days/week in either our New York City or San Ramon, CA office.
Franklin Templeton offers employees a competitive and valuable range of total rewards – monetary and non-monetary – designed to support their well-being and recognize their time, talents, and results. Along with base compensation, employees are eligible for an annual discretionary bonus, a 401(k) plan with a generous match, and recognition rewards. We also offer a comprehensive benefits package, which includes a range of competitive healthcare options, insurance, and disability benefits, employee stock investment program, learning resources, career development programs, reimbursement for certain education expenses, paid time off (vacation / holidays / sick / leave / parental & caregiving leave / bereavement / volunteering / floating holidays) and a motivational wellbeing program. We expect the annual salary for this position to range between $155,700 – $183,200, depending on location and level of relevant experience, plus discretionary bonus.
#MID_SENIOR_LEVEL
#LI-Hybrid
At Franklin Templeton, we believe your benefits should support your life, your goals, and your future. That’s why we offer a comprehensive Total Rewards package designed to help you thrive both personally and professionally.
Highlights of our benefits include
Paid Time Off: Three weeks of PTO in your first year
Health Coverage: Competitive medical, dental, and vision insurance to support your well-being
Retirement Savings: 401(k) plan with an 85% company match on pre-tax and/or Roth contributions, up to IRS limits
Equity & Investing: Employee Stock Investment Plan (ESIP) with discounted share purchase opportunities
Learning Education Assistance Program (LEAP): To support your ongoing growth and career advancement
Employee Investment Benefits: Opportunity to purchase company funds with no sales charge
Franklin Templeton is an Equal Opportunity Employer. We are committed to providing equal employment opportunities to all applicants and employees, and we evaluate qualified applicants without regard to ancestry, age, color, disability, genetic information, gender, gender identity, or gender expression, marital status, medical condition, military or veteran status, national origin, race, religion, sex, sexual orientation, and any other basis protected by federal, state, or local law, ordinance, or regulation.
Artificial intelligence or automated tools may be used to assist in the screening, assessment, or selection of applicants for this position.
