You will develop and maintain benchmarks: curated “golden sets,” scenario suites, stress/adversarial cases, and continuously refreshed market/regime-based test corpora
You will build automated quality gates and regression workflows that block releases when key metrics degrade
You will partner with engineering and product to define safe tool/action contracts (deterministic previews, confirmations, auditability) and ensure predictable assistant behavior
You will own model improvement loops tied to evals: data collection/labeling strategies, error taxonomy, prompt/tooling changes, and when appropriate, fine-tuning or preference optimization to measurably improve benchmark performance
You will design and operate monitoring + incident response for AI: telemetry, alerting, RCA, and “fix-forward” processes
You will develop a deep understanding of trading concepts (margin, shorting, portfolio margin, risk, execution) and how to express them accurately and understandably to users
Tech Stack: Rust, TypeScript, Postgres, React (Web), React Native (Mobile), observability/telemetry tooling, LLM APIs + model serving, eval/training pipelines
Requirements
At least Eight (8) years of experience shipping production software; strong proficiency with any programming language
Strong knowledge of computer science fundamentals, testing methodology, and systems design
Experience building evaluation frameworks, test harnesses, and benchmark suites for complex systems (LLMs/agents/search/retrieval/ranking/recommenders)
Experience running model improvement cycles: dataset curation, labeling/QA, offline experimentation, and deploying changes with measurable impact on benchmarks
Ability to define metrics, build measurement pipelines, and drive engineering/product decisions from data
Comfort working across the stack: debugging model/tooling failures, instrumenting services, and partnering with frontend/product on UX patterns that improve safety and trust
High degree of self-motivation and willingness to jump into unfamiliar areas to solve problems
Bonus
Experience with fine-tuning, preference optimization, distillation, or prompt/compiler-style techniques for improving tool-use reliability
Experience creating domain-specific benchmarks and adversarial suites (e.g., “known-bad” scenarios) for high-stakes applications
Deep experience with trading across asset classes, margin types, etc.
Experience with Rust and performance-sensitive services
Experience designing incident response and SLOs for ML/AI systems
We Offer
The Base Salary Range for this role is $180,000 - $250,000. This range is representative of the starting base salaries for this role at Clear Street. Where a candidate falls in this range will be based on job related factors such as relevant experience, skills, and location. This range represents Base Salary only, which is just one element of Clear Street's total compensation. The range stated does not include other factors of total compensation such as bonuses or equity.
At Clear Street, we offer competitive compensation packages, company equity, 401k matching, gender neutral parental leave, and full medical, dental and vision insurance. In-office benefits include lunch stipends, fully stocked kitchens, happy hours, a great location, and amazing views.
Our top priority is our people. We're continuously investing in a culture that promotes collaboration. We help each other through challenges and celebrate each other's successes. We believe that modern workplaces succeed by virtue of having high-performance workforces that are diverse — in ideas, in cultures, and in experiences. We are proud to be an equal opportunity employer and put in the effort to make such a workplace a daily reality. #LI-Remote