Tracing torch.cuda.empty_cache() on an RTX 4090 – Where Do the 53 MB Go?

TL;DR: After del tensor; torch.cuda.empty_cache(), PyTorch’s caching allocator still holds 53.7 MB that it won’t release. We traced the CUDA Runtime and Driver APIs with eBPF uprobes to see exactly what happens at the kernel level during the free path….


This content originally appeared on DEV Community and was authored by David Mail

TL;DR: After del tensor; torch.cuda.empty_cache(), PyTorch's caching allocator still holds 53.7 MB that it won't release. We traced the CUDA Runtime and Driver APIs with eBPF uprobes to see exactly what happens at the kernel level during the free path. The trace showed cudaFree calls hitting p99 = 1.9ms (4.6x their median) because the process keeps getting descheduled mid-free. The allocator isn't broken - the OS is interrupting it.

The Issue

pytorch/pytorch#173382 - a user calls torch.cuda.empty_cache() after deleting tensors, but GPU memory stays allocated. The caching allocator's empty_cache() only releases blocks it has marked as free, but the user sees a persistent gap between "allocated" and "reserved" memory.

Reproducing It

We loaded Qwen2.5-0.5B-Instruct, ran 3 inference rounds, and logged CUDA memory at each step. RTX 4090, PyTorch 2.10, NVIDIA driver 580.

[after model load              ] allocated=   950.2 MB  reserved=   992.0 MB  gap=    41.8 MB
[round 1: after del+empty_cache] allocated=   958.3 MB  reserved=  1012.0 MB  gap=    53.7 MB
[round 2: after del+empty_cache] allocated=   958.3 MB  reserved=  1012.0 MB  gap=    53.7 MB
[round 3: after del+empty_cache] allocated=   958.3 MB  reserved=  1012.0 MB  gap=    53.7 MB

The 53.7 MB gap stays constant across all 3 rounds. Even after deleting the model and running gc.collect(), 11.9 MB remains unreachable.

Tracing with eBPF

We attached eBPF uprobes to libcudart.so and libcuda.so to trace every CUDA memory operation, kernel launch, and synchronization call.

Five causal chains, all pointing to the same root cause:

Operation P50 P99 Slowdown
cudaMemcpyAsync 9 us 887 us 98.6x
cudaFree 413 us 1.9 ms 4.6x
cudaLaunchKernel 8 us 25 us 3.2x
cudaStreamSync 3 us 22 us 6.9x

288 context switches during the workload. Every time the Python process was descheduled, the CUDA operation in progress got delayed.

The Actual Problem

Two things stacked:

  1. PyTorch's caching allocator holds blocks for reuse by design. The 53.7 MB gap is blocks allocated at the CUDA level but not currently backing any Python tensor. empty_cache() releases only blocks the allocator has marked as truly free.

  2. The host CPU is interfering with the free path. System services (journald, atopacct, resolved) compete for CPU time. cudaFree calls take 4.6x longer at p99 because the thread gets descheduled mid-operation.

Reproduce It

The trace database is in the Ingero repo. Connect any MCP-compatible AI:

# With MiniMax via Ollama
ollmcp -m minimax-m2.7:cloud -j /tmp/ingero-mcp.json

# With Claude Code
claude --mcp-config /tmp/ingero-mcp.json

Full writeup with terminal recording: ingero.io/tracing-torch-cuda-empty-cache-rtx-4090/

Ingero is open source (Apache 2.0). GitHub


This content originally appeared on DEV Community and was authored by David Mail


Print Share Comment Cite Upload Translate Updates
APA

David Mail | Sciencx (2026-03-25T15:27:57+00:00) Tracing torch.cuda.empty_cache() on an RTX 4090 – Where Do the 53 MB Go?. Retrieved from https://www.scien.cx/2026/03/25/tracing-torch-cuda-empty_cache-on-an-rtx-4090-where-do-the-53-mb-go/

MLA
" » Tracing torch.cuda.empty_cache() on an RTX 4090 – Where Do the 53 MB Go?." David Mail | Sciencx - Wednesday March 25, 2026, https://www.scien.cx/2026/03/25/tracing-torch-cuda-empty_cache-on-an-rtx-4090-where-do-the-53-mb-go/
HARVARD
David Mail | Sciencx Wednesday March 25, 2026 » Tracing torch.cuda.empty_cache() on an RTX 4090 – Where Do the 53 MB Go?., viewed ,<https://www.scien.cx/2026/03/25/tracing-torch-cuda-empty_cache-on-an-rtx-4090-where-do-the-53-mb-go/>
VANCOUVER
David Mail | Sciencx - » Tracing torch.cuda.empty_cache() on an RTX 4090 – Where Do the 53 MB Go?. [Internet]. [Accessed ]. Available from: https://www.scien.cx/2026/03/25/tracing-torch-cuda-empty_cache-on-an-rtx-4090-where-do-the-53-mb-go/
CHICAGO
" » Tracing torch.cuda.empty_cache() on an RTX 4090 – Where Do the 53 MB Go?." David Mail | Sciencx - Accessed . https://www.scien.cx/2026/03/25/tracing-torch-cuda-empty_cache-on-an-rtx-4090-where-do-the-53-mb-go/
IEEE
" » Tracing torch.cuda.empty_cache() on an RTX 4090 – Where Do the 53 MB Go?." David Mail | Sciencx [Online]. Available: https://www.scien.cx/2026/03/25/tracing-torch-cuda-empty_cache-on-an-rtx-4090-where-do-the-53-mb-go/. [Accessed: ]
rf:citation
» Tracing torch.cuda.empty_cache() on an RTX 4090 – Where Do the 53 MB Go? | David Mail | Sciencx | https://www.scien.cx/2026/03/25/tracing-torch-cuda-empty_cache-on-an-rtx-4090-where-do-the-53-mb-go/ |

Please log in to upload a file.



There are no updates yet.
Click the Upload button above to add an update.

You must be logged in to translate posts. Please log in or register.

Related Posts