A simple guide to compile Llama.cpp and llama-cpp-python using CLBlast for older generation AMD GPUs (the ones that don't support ROCm, like RX 5500).

1. Install cmake using pip.

pip install cmake

2. Follow regular instructions to install llama-cpp-python. The uninstalled the default llama-cpp-python versions installed:

pip uninstall -y llama-cpp-python llama-cpp-python-cuda

3. Move the OpenCL folder under the C drive. Alternatively, edit the CLBlastConfig-release.cmake file to point to where you save the folder OpenCL.

   • This folder was obtained from OCL SDK Light AMD. The only difference is that that one creates an env variable to OCL_ROOT, but you can just point to the folder directly and avoid dealing with .exe files.
   • Here is the part you need to update in the CLBlastConfig-release.cmake file if you save the OpenCL folder somewhere else:

   # Import target "clblast" for configuration "Release"
   set_property(TARGET clblast APPEND PROPERTY IMPORTED_CONFIGURATIONS RELEASE)
   set_target_properties(clblast PROPERTIES
       IMPORTED_IMPLIB_RELEASE "${_IMPORT_PREFIX}/lib/clblast.lib"
       IMPORTED_LINK_INTERFACE_LIBRARIES_RELEASE "C:/OpenCL/lib/OpenCL.lib"
       IMPORTED_LOCATION_RELEASE "${_IMPORT_PREFIX}/lib/clblast.dll"
   )
   

   Edit the IMPORTED_LINK_INTERFACE_LIBRARIES_RELEASE to where you put OpenCL folder.

4. Reinstall llama-cpp-python using the following flags.The location C:\CLBlast\lib\cmake\CLBlast should be inside of where you downloaded the folder CLBlast from this repo (you can put it anywhere, just make sure you pass it to the -DCLBlast_DIR flag)

   • For powershell, set env variables first:

   $env:CMAKE_ARGS="-DLLAMA_CLBLAST=on -DCLBlast_DIR=C:\CLBlast\lib\cmake\CLBlast"
   $env:FORCE_CMAKE=1 
   pip install llama-cpp-python --no-cache-dir
   

   • If you are installing through cmd, you can just use this instead:

   CMAKE_ARGS="-DLLAMA_CLBLAST=on -DCLBlast_DIR=C:\CLBlast\lib\cmake\CLBlast" FORCE_CMAKE=1 pip install llama-cpp-python --no-cache-dir
   

5. All done, fire it up and you should see something like this when you load a model using llama-cpp-python:

1. Install cmake using pip.

pip install cmake

2. Clone llama.cpp repo

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

3. Create a build folder and run cmake with the following flags:

• -DLLAMA_CLBLAST=on to enable CLBlast
• -DCLBlast_DIR=C:\CLBlast\lib\cmake\CLBlast to point to where you downloaded the folder CLBlast from this repo (you can put it anywhere, just make sure you pass it to the -DCLBlast_DIR flag)

mkdir build
cd build
cmake .. -DLLAMA_CLBLAST=on -DCLBlast_DIR=C:/CLBlast

4. Build the project

cmake --build . --config Release

5. Add the the clblast.dll folder to your path. Or alternatively. Just copy the clblast.dll to the Release folder where you have your llama-cpp executables.

   • You can find the clblast.dll in C:\CLBlast\lib on this repo.

6. Run a model

cd bin/Release
.\main.exe -m 'C:\YOUR-MODELS-FOLDER\codellama-7b-instruct.Q5_K_S.gguf' -ngl 40 -t 6 -p 'write a python function to delete all folders that are bigger than 100 mb on current directory'

where the flags mean:

• -m path to the model
• -ngl number of layers to offload to gpu
• -t number of threads
• ./main.exe is the executable generated by cmake

Error:

Error The code execution cannot proceed because clblast.dll was not found

Solution: Put clblast.dll in the same folder as the executables. You can find the clblast.dll in C:\CLBlast\lib on this repo. Or add the folder where you have the clblast.dll to your path.

Error:

CMake Error in CMakeLists.txt:
  Imported target "clblast" includes non-existent path

    "C:/OpenCL/include"

Solution Edit the CLBlastConfig-release.cmake file to point to where you saved the folder OpenCL if it is different from C:\OpenCL.