Staff Engineer, Inference Optimizations
Job Description:
- Lead technical strategy for benchmarking and performance optimizations at inference engine and GPU kernel layers
- Engineer solutions for attention-layer optimization, memory and precision management, and advanced parallelization across multi-node GPU clusters
- Implement cutting-edge optimization techniques for generative AI infrastructure
- Improve batch-size performance using AMD's AITER library for AMD MI355X, tuning CK or ASK for FP8/BF16
- Identify kernel-fusion opportunities for GLM-5 Transformer-block kernels, including FlashAttention and RMS Norm
- Tune expert-gateway router kernels for MoE models such as Qwen3-235B, DeepSeek V3, and GLM-5
- Advise on modern NVIDIA and AMD GPU hardware procurement and software integration
- Develop and deploy quantization techniques using FP8, INT8, and experimental FP4
- Provide technical mentorship through high-quality code and design reviews without direct management
- Partner with Product Management and TPMs to translate hardware limits into shippable product features
- Contribute to and integrate open-source AI while maintaining a presence in GPU infrastructure and model-performance communities
Requirements:
- 5+ years of experience in high-performance computing or AI infrastructure
- Proven track record of solving compute utilization and memory bandwidth bottlenecks
- Deep familiarity with the Gen AI landscape, including LLM, VLM, and LMM model families
- Hands-on experience with attention-layer optimizations and parallelization strategies across distributed GPU environments
- Comprehensive understanding of NVIDIA and AMD GPU architectures and software ecosystems, including CUDA and ROCm
- Extensive experience integrating, building with, and contributing to open-source software projects
- Excellent system design skills related to low-level GPU programming, optimization, memory access patterns, and parallel execution
- Experience acting as a technical lead and driving design and delivery through cross-functional alignment and expert-level delegation
- Deep understanding of GPU architectures, including SMs, warp scheduling, and Tensor Cores
- Expert-level Triton or CUDA proficiency
- Experience contributing to the Triton compiler or writing custom CUDA kernels for a major LLM is desired by the posting's wording
Benefits:
- Reimbursement for relevant conferences, training, and education, subject to company policy
- Flexible time off
- Local team meetups
- Mental health and well-being support
- Annual bonus plan eligibility for non-sales roles, subject to plan terms and local eligibility
- Potential new hire equity grant, subject to plan terms and local eligibility
- Employee Stock Purchase Program opportunity in participating countries, subject to eligibility
- Comprehensive total rewards package
- Professional development resources and career development support