Mistral AI traces a vLLM memory leak outside the heap
Mistral AI
- Infrastructure
- #vllm
- #nixl
- #bpf trace
- #pmap
Browse Infrastructure topic page →
Mistral AI engineers describe debugging a suspected vLLM memory leak that appeared only in a prefill/decode disaggregated setup with NIXL and graph compilation, growing system memory about 400 MB per minute on production-like traffic. Standard Python profilers and Heaptrack showed a stable heap, so they turned to pmap and BPFtrace and found growing anonymous memory mappings, pointing away from glibc-managed allocations.
What the investigation found
Mistral AI engineers report that during pre-production testing of disaggregated serving with a frontier model, system memory climbed steadily under specific conditions: vLLM, the Mistral Medium 3.1 model, and graph compilation enabled. There were no errors, only roughly 400 MB per minute of growth on production-like traffic, eventually leading to out-of-memory. The leak appeared only on the decode side of a prefill/decode setup using NIXL for KV cache transfer, which pointed at that path. Python profilers such as Memray and Guppy 3 showed nothing abnormal, and Heaptrack's heap view stayed stable while peak resident memory differed between snapshots. Using pmap, the team saw some anonymous memory mappings growing with shifting start addresses, a pattern they associate with mremap or repeated munmap/mmap. That indicated allocations outside glibc's malloc, so they moved to BPFtrace for syscall-level tracing.
A scenario for an inference platform engineer
An engineer running disaggregated inference could try this sequence when a rollout shows slow memory growth without errors. First confirm the symptom is reproducible only in the disaggregated configuration, as the authors did by failing to reproduce it on a smaller model. Then record peak resident memory rather than assuming heap tools tell the whole story; the Mistral account shows Heaptrack reporting a stable heap while RSS diverged. Only after that would it make sense to inspect anonymous mappings with pmap over time. This is conditional guidance: it applies when memory grows steadily and existing profilers look clean, not as a general monitoring replacement.
A trial and how to judge it
A team could try a bounded experiment: capture a baseline RSS-over-time curve during a fixed-duration benchmark in the affected configuration, then repeat with tracer tooling attached and compare the two curves. Success would mean the growth pattern is visible during tracing and attributable to specific mappings or syscalls. Failure checks should include confirming the tracer itself does not change the leak's behavior and that the reproduction still holds. Because tracing at syscall level touches a running server, keep a human reviewer on results and avoid running unproven tracing on production traffic; the authors describe their work as investigation on a pre-production-like setup.
Trade-offs and unknowns
The approach costs time: the authors describe trying multiple profilers before reaching kernel tracing, and note that GDB crashed their setup and Valgrind was impractical. BPFtrace is presented as low-overhead and kernel-safe, but that is the publisher's characterization, not an independent measurement. The excerpt does not disclose the final root cause, the fix, or whether the issue was resolved upstream, so readers should not assume a patch exists. Teams without Linux syscall tracing experience, or running outside the described prefill/decode and NIXL arrangement, have less reason to adopt this path first.
Context and limitations
- Based on a short excerpt of a longer engineering post; the root cause and fix are not visible in the supplied text.
- The 400 MB per minute figure is the authors' observation on their own pre-production setup, not an independent benchmark.
- Tools and commands are described for the authors' environment; applicability to other configurations is untested here.
- Based on bounded source text, not independent testing. Practical suggestions are editorial proposals.