Ollama not using the GPU on Windows: how to check and fix it

How to tell in 30 seconds whether it uses the graphics card

Run a short question with the --verbose option, which prints the generation speed (“eval rate”) at the end. Without closing the model, run ollama ps in another window. According to Ollama’s documentation, the PROCESSOR column shows where the model was loaded, and “100% GPU” means it’s entirely on the graphics card (Ollama FAQ).

Then check nvidia-smi: if the card’s used memory doesn’t rise when the model loads, the model isn’t on the graphics card. In the CPU-only test, VRAM didn’t move (a peak of 2,805 MiB against about 3,000 at rest) and Ollama’s API reported 0 GB on the GPU.

How to fix it, cause by cause

Go in this order: first whatever stops the card from being seen, then whatever only stops the model from fitting. After each change, fully close Ollama (Get-Process ollama* | Stop-Process -Force) and open it again, because variables are only read at startup.

• Outdated driver: Ollama works with NVIDIA cards with compute capability 5.0 or higher and driver 550 or later; cards from 5.0 to 6.2 need driver 570 (GPU support in the Ollama documentation). Check your version with nvidia-smi and update from NVIDIA’s website if needed.
• A variable hides the card: Get-ChildItem Env: | Where-Object Name -match "CUDA|HIP|GGML|OLLAMA". According to the documentation, CUDA_VISIBLE_DEVICES with an invalid value such as -1 forces the CPU, and OLLAMA_LLM_LIBRARY can force the CPU library. To delete it from your user: [Environment]::SetEnvironmentVariable("CUDA_VISIBLE_DEVICES", $null, "User").
• The model has num_gpu set to 0: check with ollama show <model> --parameters. If you see num_gpu 0 (exactly what we used in the tests to simulate a PC without a graphics card), create the model without that line or pass another value in the request.
• The model doesn’t fit: if PROCESSOR shows a split, the model and its context exceed the free VRAM. Try a smaller model or lower num_ctx. With 16 GB, the 4B–12B models fit whole; the 35B one didn’t.
• Another program takes the VRAM: with three models loaded and a game open, VRAM reached 15.6 of 16 GB and gemma4:12b dropped from 111 to about 3.8 tokens per second. Unload what you don’t use with ollama stop <model>; by default, Ollama keeps each model in memory for 5 minutes.

How to check that it’s fixed

Repeat the 30-second test and look at three things at once: PROCESSOR at “100% GPU”, a speed similar to the list below, and VRAM in nvidia-smi rising when the model loads. Don’t rely on just one. With gemma4:12b, Ollama’s API (/api/ps, the same one ollama ps queries) reported a size of 1.15–1.5 GB, while the log said 7.0 GB on the GPU and nvidia-smi showed another 8.3–9.3 GB. The GPU percentage was correct; the size was not. All the data is in our speed and VRAM test of local models.

Another trap when measuring: if you repeat exactly the same question, the reading jumps to 8,000–13,000 tokens per second because Ollama reuses what it already processed. Change the text or look at the first run.

How much it changes: with the graphics card and without it

To simulate a PC without a graphics card, we forced num_gpu 0 on a test PC with an Intel Core Ultra 5 245K, about 31 GB of RAM and a 16 GB graphics card, using the same 524-token question. If your figures look like the CPU ones, you know what’s happening. Keep in mind that it’s a recent desktop processor; on a laptop it will be even slower.

• qwen3.5:4b: 92–95 tokens per second (t/s) with the graphics card and 11.9 t/s on the CPU alone, about 8 times slower; it reads the question at 750 t/s instead of 121.
• qwen3.5:9b: 56–58 t/s with the graphics card and 5.7 on the CPU alone, about 10 times slower; it reads at 1,760 t/s instead of 58.
• Graphics card memory each model used: qwen3.5:4b, 3.6–4.6 GB; qwen3.5:9b, 6.4–7.5 GB; gemma4:12b, 8.3–9.3 GB.
• qwen3.6:35b (22 GB) doesn’t fit in 16 GB: Ollama split it into 12.4 GB on the graphics card and 8.3 GB of RAM, and it ended in a 500 error (we cover it in Ollama error 500 “unable to load model”).

Going deeper: which graphics card Ollama detects at startup

ollama ps tells you what’s happening; the log tells you why. With the app, the log is at %LOCALAPPDATA%\Ollama\server.log (Ollama troubleshooting). On startup, Ollama writes an “inference compute” line for each graphics card it detects, with the library (CUDA), the name, the driver and the free memory. If that line doesn’t name your card, Ollama doesn’t see it, and nothing you change in the model will fix that. When a model loads, the “offloaded” line says how many layers went to the graphics card.

The “server config” line shows the variables Ollama read, including CUDA_VISIBLE_DEVICES. On our test PC it was empty, which is correct.

Rare cases

• AMD graphics cards: on Windows, Ollama asks for the ROCm v7 drivers, and the documentation only names the Radeon RX 7600 to 7900 and the PRO W7500 to W7900; for the rest, Vulkan support is available if it’s installed. We haven’t tested this.
• Laptop with integrated and dedicated graphics: check that the “inference compute” line names the dedicated one. We haven’t tested this on a laptop.
• The VRAM is taken even when you have nothing open: the Windows desktop and other programs already used about 2.9–3.0 GB at rest on our test PC. Count them when you choose a model.
• LM Studio doesn’t use the GPU either: there the equivalent setting is the number of layers sent to the graphics card (GPU offload) when the model loads. We didn’t include LM Studio in these tests.