Oversee fine-tuning/training workflows: dataset curation, annotation quality, training configurations, and generalization across document variants.
Build and maintain evaluation infrastructure: benchmark/golden datasets, regression suites, and automated scoring to catch regressions pre-production.
Define confidence calibration and escalation logic so systems estimate uncertainty and route low-confidence outputs to human reviewers with the right context.
Partner with governance/model risk and data engineering: produce validator-ready documentation (explainability/auditability) and ensure robust, refreshed data/annotation/eval pipelines; drive AI-native development practices to improve velocity.
Required qualifications, capabilities and skills
10+ years in NLP/AI/ML or computational linguistics, including 3+ years leading technical teams with direct reports.
Hands-on LLM internals expertise: tokenization impacts, attention limits, context window management, and temperature/sampling trade-offs.
Proven prompt architecture design/debugging: multi-turn chains, few-/many-shot, chain-of-thought, self-consistency, and constitutional AI.
Strong RAG system design experience: embeddings trade-offs (e.g., ada/BGE/Cohere), chunking for semi-structured docs, hybrid retrieval (dense+sparse), and re-ranking.
Fine-tuning experience: LoRA/QLoRA, instruction tuning, RLHF/DPO, dataset curation, and evaluating tuned vs prompted performance.
Python + ML engineering proficiency: PyTorch, Hugging Face, LangChain/LlamaIndex (or equivalents), vector DBs (Pinecone/Weaviate/pgvector), and API development.
AI evaluation systems experience beyond F1: faithfulness, answer relevance, RAG context precision/recall, automated eval pipelines, and LLM-as-judge.
Deep understanding of LLM failure modes: hallucinations, sycophancy, long-context instruction degradation, prompt-format sensitivity, and catastrophic forgetting.
Structured output enforcement in production: JSON mode, function calling, constrained decoding, output parsers, and schema validation.
Build-vs-buy/model selection track record: benchmarking foundation models (GPT-4/Claude/Llama/Mistral) against task requirements.
Leadership under ambiguity: pragmatic trade-offs, rapid iteration as practices evolve, and strong communication with compliance, risk, and business stakeholders.
Preferred qualifications, capabilities and skills
Domain expertise in KYC/AML or financial document processing: entity extraction from registries, beneficial ownership structures, sanctions screening logic, adverse media classification.
Experience designing autonomous AI agents: tool-use patterns, planning/reasoning loops, memory architectures, and safety guardrails for regulated environments.
Knowledge of AI security/adversarial robustness: prompt injection defense, jailbreak detection, data poisoning awareness, and output monitoring for sensitive financial data.
Experience with model distillation to produce smaller, faster models for cost-effective deployment.
Familiarity with AI observability/monitoring: tracking prompt/model performance, drift detection, alerting, and health dashboards.
Experience with multi-modal AI combining OCR, layout understanding, and LLM-based extraction for complex documents.
Advanced degree (MS/PhD) in CS/NLP/ML or equivalent depth via publications, open-source, or production system design and Passion for talent development: growing engineers from junior prompt writers into senior AI system designers via structured mentorship.