If you're running local AI models on Windows and wondering why they feel sluggish, the operating system itself might be the bottleneck.
A developer running Ollama on identical hardware - an RTX 8000 with 48GB VRAM (Turing architecture), Intel Core i9-9900K, and 64GB DDR4 RAM - found that tokens per second (the speed at which AI models generate text) was roughly twice as fast on Ubuntu 22.04 compared to Windows 10. Same machine, same Ollama version, same models. The only variable was the OS.
This isn't entirely surprising. Linux has long had an edge in GPU compute workloads because its CUDA driver stack (NVIDIA's software layer for GPU computing) runs with less overhead. Windows adds layers of abstraction through WDDM (Windows Display Driver Model) that are designed for desktop graphics but add latency for pure compute tasks. Linux's more direct hardware access means less gets in the way between the model and the GPU.
For casual use, the difference might not matter much. But if you're running a local model as part of a workflow - coding assistants, document processing, batch summarization - a 2x speed difference adds up fast. A response that takes 8 seconds on Windows could come back in 4 on Linux.
The practical takeaway: if you have a dedicated machine for local AI inference (inference means running a trained model to generate outputs, as opposed to training it), Linux is the clear choice. Even WSL2 (Windows Subsystem for Linux) can close some of this gap, though native Linux still wins on raw GPU throughput. For anyone building a home lab or local AI server, Ubuntu or a similar Linux distribution should be the default starting point.