young and beautiful
Describe the bug. I&39;m using pytorchlightning with deepspeed to train a ddpm model, the training started with 6G GPU memory used, but it keeps increasing as the training continues, after about 100 steps the memory usage increased to more than 11G, causing CUDA out of memory issue on my 2080ti card. 2022. 6. 25.. 22. 183; CUDA - Memories . Apart from the device DRAM, CUDA supports several additional types of memory that can be used to increase the CGMA ratio for a kernel. We know that accessing the DRAM is slow and expensive. To overcome this problem, several low-capacity, high-bandwidth memories , both on-chip and off-chip
are present on a CUDA GPU. Mar 15, 2021 Image size 224, batch size 1. RuntimeError CUDA out of memory. Tried to allocate 1.91 GiB (GPU 0; 24.00 GiB total capacity; 894.36 MiB already
allocated; 20.94 GiB free; 1.03 GiB reserved in total by PyTorch). Even with stupidly low image sizes and batch sizes. EDIT SOLVED - it was a number of workers problems, solved it by ..