GPUs Supported by Msty

Nvidia GPUs

Msty (and Ollama, the underlying Local AI engine) supports Nvidia GPUs with compute capability 5.0+.

Check your compute compatibility to see if your card is supported: https://developer.nvidia.com/cuda-gpus

GPU Selection

If you have multiple NVIDIA GPUs in your system and want to limit Msty to use a subset, you can set CUDA_VISIBLE_DEVICES to a comma separated list of GPUs. This value can be set in Settings -> General Settings -> Local AI > Service Configurations > Advanced Configurations. Numeric IDs may be used, however ordering may vary, so UUIDs are more reliable. You can discover the UUID of your GPUs by running nvidia-smi -L If you want to ignore the GPUs and force CPU usage, use an invalid GPU ID (e.g., "-1")

You can pass extra environment variables to the Local AI service

Laptop Suspend Resume

On Linux, after a suspend/resume cycle, sometimes Msty/Ollama will fail to discover your NVIDIA GPU, and fallback to running on the CPU. You can work around this driver bug by reloading the NVIDIA UVM driver with sudo rmmod nvidia_uvm && sudo modprobe nvidia_uvm

AMD Radeona GPUs

Msty/Ollama supports the following AMD GPUs:

Linux Support

Windows Support

With ROCm v6.1, the following GPUs are supported on Windows.

Overrides on Linux

Msty/Ollama leverages the AMD ROCm library, which does not support all AMD GPUs. In some cases you can force the system to try to use a similar LLVM target that is close. For example The Radeon RX 5400 is gfx1034 (also known as 10.3.4) however, ROCm does not currently support this target. The closest support is gfx1030. You can use the environment variable HSA_OVERRIDE_GFX_VERSION with x.y.z syntax. So for example, to force the system to run on the RX 5400, you would set HSA_OVERRIDE_GFX_VERSION="10.3.0" as an environment variable for the server. If you have an unsupported AMD GPU you can experiment using the list of supported types below.

At this time, the known supported GPU types on Linux are the following LLVM Targets. This table shows some example GPUs that map to these LLVM targets:

AMD is working on enhancing ROCm v6 to broaden support for families of GPUs in a future release which should increase support for more GPUs.

GPU Selection

If you have multiple AMD GPUs in your system and want to limit Msty/Ollama to use a subset, you can set HIP_VISIBLE_DEVICES to a comma separated list of GPUs. This value can be set in Settings -> General Settings -> Local AI > Service Configurations > Advanced Configurations. You can see the list of devices with rocminfo. If you want to ignore the GPUs and force CPU usage, use an invalid GPU ID (e.g., "-1")

You can pass extra params to a model globally

You can also pass main_gpu parameter to each model to specify which GPU to use for that model. This can be set globally for all models in Settings -> General Settings -> Local AI > Model Configurations > Advanced Configurations or per model in the model's configuration available when chatting with the model. This value can even be set per chat.

You can pass extra params (A) per chat (B) or per model (B)

Container Permission

In some Linux distributions, SELinux can prevent containers from accessing the AMD GPU devices. On the host system you can run sudo setsebool container_use_devices=1 to allow containers to use devices.

Metal (Apple GPUs)

Msty/Ollama supports GPU acceleration on Apple devices via the Metal API.

Windows Troubleshooting

Check out the AMD ROCm on Windows Issues guide for common issues and solutions when using AMD GPUs on Windows.