Develop high-performance control-plane and data-plane services in Go and performance-critical components (tokenizers, schedulers, runtimes, gRPC/HTTP gateways, native extensions) in Rust.
Optimize GPU inference for throughput, latency, and cost using continuous batching, KV-cache management, quantization (INT8/FP8/AWQ/GPTQ), speculative decoding, tensor/pipeline parallelism, and efficient memory use.
Integrate and extend inference engines such as vLLM, TensorRT-LLM, Triton Inference Server, Ray Serve, KServe, SGLang, llama.cpp, and ONNX Runtime; contribute fixes upstream where it makes sense.
Build and manage GPU cluster infrastructure on Kubernetes (NVIDIA GPU Operator, device plugins, MIG/time-slicing, node pools, scheduling and bin-packing) across cloud and on-prem environments.
Own the full delivery path: CI/CD pipelines, model packaging and registries, canary and blue/green rollouts, infrastructure as code, and automated performance regression testing.
Build AI observability into everything: metrics, logs, and traces for token throughput, time-to-first-token, GPU utilization, queue depth, error and hallucination signals, using OpenTelemetry, Prometheus, and Grafana.
Provide clean APIs, SDKs, and developer tooling (OpenAI-compatible endpoints, internal services, and lightweight UIs or dashboards) so AI engineers and agent builders can deploy and monitor models self-serve.
Support agentic and RAG workloads by serving embedding, reranking, and generation models alongside the Enterprise Vector Store, with predictable latency and graceful degradation.
Drive reliability, security, and cost efficiency: SLOs, load and chaos testing, capacity planning, GPU cost attribution, and secure model and tenant isolation.
Who You’ll Work With
You will collaborate daily with AI researchers, ML engineers, data scientists, and platform and SRE teams who champion diversity, equity, and inclusion as fundamental to our success. You’ll be part of a cohesive team delivering high-quality, highly visible AI/ML functionality within the Teradata Vantage platform. You will report to the Sr. Manager, Software Engineering, who will champion your growth and empower your contributions.
What Makes You a Qualified Candidate
3+ years of professional software engineering experience in backend systems, distributed systems, or AI/ML infrastructure.
Strong, production-level proficiency in Go (concurrency, gRPC, profiling, service design) and working proficiency, or a demonstrated passion for learning, Rust (ownership, async runtimes such as Tokio, FFI, performance tuning).
Hands-on experience with GPU computing for ML: CUDA fundamentals, GPU profiling (Nsight, nvidia-smi/DCGM), memory and bandwidth tradeoffs, and multi-GPU or multi-node inference.
Experience serving LLMs or generative models in production, with practical knowledge of inference optimization techniques and the serving frameworks listed above.
Solid understanding of transformer architectures and the LLM inference lifecycle (prefill vs. decode, attention, KV cache, sampling), plus familiarity with PyTorch and Hugging Face tooling.
Strong system design skills and a deep understanding of distributed systems, networking, concurrency, and failure modes.
Experience with containers and orchestration (Docker, Kubernetes, Helm) and CI/CD (GitHub Actions, Jenkins, Argo CD, or similar), plus infrastructure as code (Terraform).
Full-stack engineering experience: comfortable building REST/gRPC APIs and a pragmatic front end (React or similar) for dashboards, playgrounds, or admin tools; Python for glue, tooling, and ML integration.
Experience with observability practices and tools (OpenTelemetry, Prometheus, Grafana, tracing) applied to ML or LLM workloads.
You are obsessive about reliability, debuggability, and ensuring AI systems behave predictably and deterministically when needed.
You code daily, own systems end to end, and are knowledgeable about open-source tools and know how to leverage and extend them.
What You Bring
A Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field, or equivalent practical experience.
Genuine excitement for AI, LLMs, and the systems that make them efficient and affordable to run at scale.
Strong communication skills and a bias toward measuring, benchmarking, and iterating.
Familiarity with modern data platforms such as Teradata, Snowflake, or Databricks is a plus.
Bonus Points
Experience writing or tuning CUDA / Triton (OpenAI) / CUTLASS kernels, or contributing to inference engines.
Experience with on-device or edge ML inference (ONNX, TFLite, llama.cpp, quantized small language models), embedded or mobile deployment.
Knowledge of RAG pipelines, vector databases, embeddings, and agent frameworks (LangChain, LlamaIndex, ReAct-style agents).
Experience with model monitoring, drift detection, evaluation harnesses, and responsible AI practices.
Open-source contributions, technical writing, or public projects in ML systems, Go, or Rust.
Experience with service meshes, RDMA/InfiniBand/NCCL, Slurm, or Ray clusters.
#LI-VB1
