Background
OpenCL (Open Computing Language) is an open, royalty-free standard for cross-platform, parallel programming of diverse accelerators found in supercomputers, cloud servers, personal computers, mobile devices and embedded platforms. OpenCL specifies a programming language (based on C99) for programming these devices and application programming interfaces (APIs) to control the platform and execute programs on the compute devices. Similar to CUDA, OpenCL has been widely used to program GPUs and is supported by most GPU vendors.
Llama.cpp + OpenCL
The llama.cpp OpenCL backend is designed to enable llama.cpp on Qualcomm Adreno GPU firstly via OpenCL. Thanks to the portabilty of OpenCL, the OpenCL backend can also run on certain Intel GPUs such as those that do not have
support although the performance is not optimal.
OS
OSStatusVerifiedAndroidSupportSnapdragon 8 Gen 3, Snapdragon 8 EliteWindowsSupportWindows 11 Arm64 with Snapdragon X EliteLinuxSupportUbuntu 22.04 WSL2 with Intel 12700HHardware
Adreno GPU
Verified devices
Adreno GPUStatusAdreno 750 (Snapdragon 8 Gen 3)SupportAdreno 810 (Snapdragon 7s Gen 3)SupportAdreno 830 (Snapdragon 8 Elite)SupportAdreno 840 (Snapdragon 8 Elite Gen 5)SupportAdreno X1-85 (Snapdragon X Elite)SupportAdreno X2-90 (Snapdragon X2 Elite)SupportA6x GPUs with a recent driver and compiler are supported; they are usually found in IoT platforms. However, A6x GPUs in phones are likely not supported due to the outdated driver and compiler.
DataType Supports
DataTypeStatusQ1_0SupportQ4_0SupportQ4_1SupportQ5_0SupportQ5_1SupportQ8_0SupportQ4_KSupportQ5_KSupportQ6_KSupportMXFP4SupportIQ4_NLSupportModel Preparation
Since common quantizations are supported now, it is recommanded to download GGUF models directly from Huggingface.
Binary Kernel Library
A prebuilt binary kernel library has been introduced for Adreno GPUs. It currently targets X2 GPUs (X2-90, X2-85 and X2-45) found in Snapdragon X2 SoC. The library currently contains kernels for MUL_MAT_ID with Q4_0, Q4_1, Q4_K, MXFP4. The library must be manually downloaded from
https://softwarecenter.qualcomm.com/catalog/item/Adreno_Kernel_Library_GGML
.
To allow using the kernel library, add -DGGML_OPENCL_USE_ADRENO_BIN_KERNELS=ON when configuring with CMake. Then, extract adreno-opencl-kernels.dll from the zip file downloaded from the above URL and put it alongside the executables. If kernels compatible with the current GPU are found in the library, they will be loaded and used.
CMake Options
The OpenCL backend has the following CMake options that control the behavior of the backend.
CMake optionsDefault valueDescriptionGGML_OPENCL_EMBED_KERNELSONEmbed OpenCL kernels into the executable.GGML_OPENCL_USE_ADRENO_KERNELSONUse kernels optimized for Adreno.GGML_OPENCL_USE_ADRENO_BIN_KERNELSOFFAllow using binary kernel lib for Adreno.Program Binary Cache
Compiled cl_program binaries are cached on disk, so subsequent runs skip the expensive compile-from-source step when nothing relevant has changed (kernel source, compile options, device, driver, or platform version).
The cache is controlled with the GGML_OPENCL_KERNEL_CACHE_DIR environment variable:
ValueBehaviorunset / empty / 1 / defaultEnabled in the platform default cache directory: %LOCALAPPDATA%\llama.cpp\cl-cache (Windows), ~/Library/Caches/llama.cpp/cl-cache (macOS), <temp dir>/llama.cpp/cl-cache elsewhere.0 / off / none / disable(d)Disabled.any other valueUsed verbatim as the cache directory path.If the chosen directory cannot be created or used, the cache disables itself for the process and kernels are compiled from source as usual. Set GGML_OPENCL_KERNEL_CACHE_DEBUG=1 to print a HIT/MISS/SAVE trace to stderr.
Android
Ubuntu 22.04 is used for targeting Android. Make sure the following tools are accessible from command line,
Git
CMake 3.29
Ninja
Python3
I. Setup Environment
Install NDK
cd~ wget https://dl.google.com/android/repository/commandlinetools-linux-8512546_latest.zip && \ unzip commandlinetools-linux-8512546_latest.zip && \ mkdir -p ~/android-sdk/cmdline-tools && \ mv cmdline-tools latest && \ mv latest ~/android-sdk/cmdline-tools/ && \ rm -rf commandlinetools-linux-8512546_latest.zip yes |~/android-sdk/cmdline-tools/latest/bin/sdkmanager "ndk;26.3.11579264"Install OpenCL Headers and Library
mkdir -p ~/dev/llm cd~/dev/llm git clone https://github.com/KhronosGroup/OpenCL-Headers && \ cd OpenCL-Headers && \ cp -r CL ~/android-sdk/ndk/26.3.11579264/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include cd~/dev/llm git clone https://github.com/KhronosGroup/OpenCL-ICD-Loader && \ cd OpenCL-ICD-Loader && \ mkdir build_ndk26 &&cd build_ndk26 && \ cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_TOOLCHAIN_FILE=$HOME/android-sdk/ndk/26.3.11579264/build/cmake/android.toolchain.cmake \ -DOPENCL_ICD_LOADER_HEADERS_DIR=$HOME/android-sdk/ndk/26.3.11579264/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include \ -DANDROID_ABI=arm64-v8a \ -DANDROID_PLATFORM=24 \ -DANDROID_STL=c++_shared && \ ninja && \ cp libOpenCL.so ~/android-sdk/ndk/26.3.11579264/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/aarch64-linux-androidII. Build llama.cpp
cd~/dev/llm git clone https://github.com/ggml-org/llama.cpp && \ cd llama.cpp && \ mkdir build-android &&cd build-android cmake .. -G Ninja \ -DCMAKE_TOOLCHAIN_FILE=$HOME/android-sdk/ndk/26.3.11579264/build/cmake/android.toolchain.cmake \ -DANDROID_ABI=arm64-v8a \ -DANDROID_PLATFORM=android-28 \ -DBUILD_SHARED_LIBS=OFF \ -DGGML_OPENCL=ON ninjaWindows 11 Arm64
A Snapdragon X Elite device with Windows 11 Arm64 is used. Make sure the following tools are accessible from command line,
Git
CMake 3.29
Clang 19
Ninja
Visual Studio 2022
Powershell 7
Python
Visual Studio provides necessary headers and libraries although it is not directly used for building. Alternatively, Visual Studio Build Tools can be installed instead of the full Visual Studio.
Note that building using Visual Studio's cl compiler is not supported. Clang must be used. Clang depends on libraries provided by Visual Studio to work. Therefore, Visual Studio must be installed. Alternatively, Visual Studio Build Tools can be installed instead of the full Visual Studio.
Powershell 7 is used for the following commands. If an older version of Powershell is used, these commands may not work as they are.
I. Setup Environment
Install OpenCL Headers and Library
mkdir -p ~/dev/llm cd ~/dev/llm git clone https://github.com/KhronosGroup/OpenCL-Headers && cd OpenCL-Headers mkdir build && cd build cmake ..-G Ninja `-DBUILD_TESTING=OFF `-DOPENCL_HEADERS_BUILD_TESTING=OFF `-DOPENCL_HEADERS_BUILD_CXX_TESTS=OFF `-DCMAKE_INSTALL_PREFIX="$HOME/dev/llm/opencl" cmake --build .--target install cd ~/dev/llm git clone https://github.com/KhronosGroup/OpenCL-ICD-Loader && cd OpenCL-ICD-Loader mkdir build && cd build cmake ..-G Ninja `-DCMAKE_BUILD_TYPE=Release `-DCMAKE_PREFIX_PATH="$HOME/dev/llm/opencl"`-DCMAKE_INSTALL_PREFIX="$HOME/dev/llm/opencl" cmake --build .--target installII. Build llama.cpp
mkdir -p ~/dev/llm cd ~/dev/llm git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp mkdir build && cd build cmake ..-G Ninja `-DCMAKE_TOOLCHAIN_FILE="$HOME/dev/llm/llama.cpp/cmake/arm64-windows-llvm.cmake"`-DCMAKE_BUILD_TYPE=Release `-DCMAKE_PREFIX_PATH="$HOME/dev/llm/opencl"`-DBUILD_SHARED_LIBS=OFF `-DGGML_OPENCL=ON ninjaLinux
The two steps just above also apply to Linux. When building for linux, the commands are mostly the same as those for PowerShell on Windows, but in the second step they do not have the -DCMAKE_TOOLCHAIN_FILE parameter, and then in both steps the backticks are replaced with back slashes.
If not installed already, install Git, CMake, Clang, Ninja and Python, then run in the terminal the following:
I. Setup Environment
Install OpenCL Headers and Library
mkdir -p ~/dev/llm cd~/dev/llm git clone https://github.com/KhronosGroup/OpenCL-Headers &&cd OpenCL-Headers mkdir build &&cd build cmake .. -G Ninja \ -DBUILD_TESTING=OFF \ -DOPENCL_HEADERS_BUILD_TESTING=OFF \ -DOPENCL_HEADERS_BUILD_CXX_TESTS=OFF \ -DCMAKE_INSTALL_PREFIX="$HOME/dev/llm/opencl" cmake --build . --target install cd~/dev/llm git clone https://github.com/KhronosGroup/OpenCL-ICD-Loader &&cd OpenCL-ICD-Loader mkdir build &&cd build cmake .. -G Ninja \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_PREFIX_PATH="$HOME/dev/llm/opencl" \ -DCMAKE_INSTALL_PREFIX="$HOME/dev/llm/opencl" cmake --build . --target installII. Build llama.cpp
mkdir -p ~/dev/llm cd~/dev/llm git clone https://github.com/ggml-org/llama.cpp &&cd llama.cpp mkdir build &&cd build cmake .. -G Ninja \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_PREFIX_PATH="$HOME/dev/llm/opencl" \ -DBUILD_SHARED_LIBS=OFF \ -DGGML_OPENCL=ON ninjaKnown Issues
Flash attention does not always improve performance.
Currently OpenCL backend works on A6xx GPUs with recent drivers and compilers (usually found in IoT platforms). However, it does not work on A6xx GPUs found in phones with old drivers and compilers.
TODO
Improve flash attention
Improve OpenCL C kernels performance