Setting Up Ollama with AMD Radeon RX 580 GPU Support Using Docker Containers

Hey everyone! I'm super excited to share with you how I set up Ollama with AMD Radeon RX 580 GPU support using Docker containers. A big shoutout to everyone whose posts and guides I've learned from along the way - your help has been invaluable!

In this post, I'll guide you through the process step by step. We'll use the following images:

1. docker.io/rocm/dev-ubuntu-22.04:5.7.1-complete - This image contains the ROCM 5.7.1 libraries, which support the RX 580 GPU.
2. mnccouk/ollama-gpu-rx580:latest - This image contains Ollama, which is compatible with ROCM 5.7.1.

Step 1: Stop the Existing Ollama Container

If you have an existing Ollama container running on your CPU, stop it to avoid conflicts:

docker stop ollama

Step 2: Start the ROCM Host Container

First, we'll start the ROCM host container, which will provide the necessary ROCM 5.7.1 libraries:

docker run -it --name rocm_host --device=/dev/kfd --device=/dev/dri docker.io/rocm/dev-ubuntu-22.04:5.7.1-complete

Inside the rocm_host container, verify that the GPU is recognized by running:

rocminfo

Step 3: Start the Ollama GPU Container

Next, start the Ollama GPU container using the volume from the existing Ollama container (if you want to use the existing models) and the ROCM host container:

docker run -d --name ollama_gpu --volumes-from rocm_host -v ollama:/root/.ollama -e HIP_PATH=/opt/rocm/lib/ -e LD_LIBRARY_PATH=/opt/rocm/lib --device /dev/kfd --device /dev/dri -p 11434:11434 mnccouk/ollama-gpu-rx580:latest

Step 4: Start a Chat Session with Ollama

Once the container is running, you can start a chat session using the llama3.1 model:

docker exec -it ollama_gpu ollama run llama3.1

Step 5: Managing the Containers

To stop the running Ollama GPU container:

docker container stop ollama_gpu

To start the Ollama GPU container again:

docker container start ollama_gpu

Step 6: Monitoring GPU Usage

To monitor GPU usage on the host machine, you can use the rocm-smi command:

rocm-smi

To monitor GPU usage inside the rocm_host container, first enter the container:

docker exec -it rocm_host bash

Then, run the rocm-smi command inside the container:

rocm-smi

Additional Notes:

• If you encounter error code 127, ensure that the rocm-smi command is correctly installed and accessible in the rocm_host container.
• If you encounter error code 137, it might be due to the container reaching its memory limit or being forcefully stopped by the system. You can adjust the container's memory and CPU limits when starting it.

System Information:

• Operating System: Debian 12
• CPU: AMD Ryzen 5500

By following these steps, you should be able to set up and run Ollama with AMD Radeon RX 580 GPU support using Docker containers. Happy computing, and thank you again to everyone who helped me along the way!

Cheers,
Levent Sunay