Senior Software Engineer - Inference Engine

ABOUT THE JOB

Software Engineer (Inference Engine) is responsible for developing and optimizing a high-performance inference engine for Large Language Models (LLMs) and multimodal LLMs running on FuriosaAI NPUs.

In this role, you will proactively research and apply the state-of-the-art inference optimization techniques to our inference engine. You will work in close collaboration with the compiler and hardware teams to enhance the engine's performance to its full potential.

RESPONSIBILITIES

• Design and implement FuriosaAI’s next-generation inference engine for large and multimodal language models—comparable in capability to frameworks such as vLLM and SGLang—optimized for throughput, latency, and memory efficiency.

• Design and implement advanced inference optimizations—such as speculative decoding, KV-cache management, tensor/model parallelism, memory-efficient execution, and scheduling—in our production inference engine.

• Design and develop capabilities for distributed and scalable inference, including prefill–decode (PD) and encode–prefill–decode (EPD) disaggregation, disaggregated speculative decoding, and hierarchical and external KV-cache storage such as HiCache and Mooncake.

• Collaborate closely with the Compiler team to co-design and optimize execution for FuriosaAI NPUs, improving system-level throughput, latency, and memory utilization.

• Proactively research, evaluate, and integrate state-of-the-art inference optimization techniques and key features of LLM serving frameworks into our production inference engine.

MINIMUM QUALIFICATIONS

• BS degree in Computer Science, Engineering, or a related field, with at least 3 years of relevant industry experience, or equivalent practical experience

• Proficiency in Rust or C++ programming skill

• Knowledge and passion of deep learning, LLM, and/or generative AI models

• Excellent problem-solving and data analysis skills.

• Strong communication and collaboration skills.

PREFERRED QUALIFICATIONS

• Experience in building inference serving systems for large models, encompassing batching, scheduling, caching, and load balancing.

• A deep understanding of performance optimization systems.

• Proficiency in C++/CUDA or Triton kernel development

• Contributions to open-source inference frameworks such as vLLM, SGLang, or TensorRT-LLM.

CONTACT

• [email protected]