114 lines
2.9 KiB
Markdown
114 lines
2.9 KiB
Markdown
# RKNN
|
|
|
|
##### Installing RKNN LLM and RKNN Toolkit 2
|
|
|
|
Réf. : [https://github.com/Pelochus/ezrknpu](https://github.com/Pelochus/ezrknpu)
|
|
|
|
```bash
|
|
sudo apt install cmake g++ make libgl1 libglx-mesa0 -y
|
|
|
|
cd /mnt/data
|
|
sudo curl https://raw.githubusercontent.com/Pelochus/ezrknpu/main/install.sh | sudo bash
|
|
```
|
|
|
|
##### Test run
|
|
|
|
```bash
|
|
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/Pelochus/qwen-1_8B-rk3588 # Running git lfs pull after is usually better
|
|
cd qwen-1_8B-rk3588 && git lfs pull # Pull model
|
|
rkllm qwen-chat-1_8B.rkllm # Run!
|
|
```
|
|
|
|
---
|
|
|
|
##### Installation of Miniforge3
|
|
|
|
```bash
|
|
wget -c https://github.com/conda-forge/miniforge/releases/download/24.9.0-0/Miniforge3-24.9.0-0-Linux-aarch64.sh
|
|
|
|
chmod 777 Miniforge3-24.9.0-0-Linux-aarch64.sh
|
|
|
|
bash Miniforge3-24.9.0-0-Linux-aarch64.sh
|
|
```
|
|
|
|
##### Create RKLLM-Toolkit Conda Environment
|
|
|
|
```bash
|
|
source ~/miniforge3/bin/activate
|
|
|
|
conda create -n RKLLM-Toolkit python=3.8
|
|
|
|
#
|
|
# To activate this environment, use
|
|
#
|
|
# $ conda activate RKLLM-Toolkit
|
|
#
|
|
# To deactivate an active environment, use
|
|
#
|
|
# $ conda deactivate
|
|
|
|
conda activate RKLLM-Toolkit
|
|
```
|
|
|
|
##### Install RKLLM-Toolkit
|
|
|
|
```bash
|
|
pip3 install rkllm_toolkit-x.x.x-cp38-cp38-linux_x86_64.whl
|
|
```
|
|
|
|
##### Install OpenCL
|
|
|
|
Download the ARM Mali GPU blob from rockchip's repository and put it into /usr/lib/ as follows. And install the firmware for the GPU if not already installed.
|
|
|
|
```bash
|
|
cd /usr/lib && sudo wget https://github.com/JeffyCN/mirrors/raw/libmali/lib/aarch64-linux-gnu/libmali-valhall-g610-g6p0-x11-wayland-gbm.so
|
|
cd /lib/firmware && sudo wget https://github.com/JeffyCN/mirrors/raw/libmali/firmware/g610/mali_csffw.bin
|
|
```
|
|
|
|
Add the Mali GPU blob to the OpenCL ICD config file as follows;
|
|
|
|
```
|
|
sudo apt install mesa-opencl-icd clinfo
|
|
```
|
|
|
|
On Ubuntu, you may get not found errors, especially huawei links. Ignore them. It looks OK. Proceed with the followings;
|
|
|
|
```bash
|
|
sudo mkdir -p /etc/OpenCL/vendors
|
|
echo "/usr/lib/libmali-valhall-g610-g6p0-x11-wayland-gbm.so" | sudo tee /etc/OpenCL/vendors/mali.icd
|
|
```
|
|
|
|
Set the dependencies of the Mali OpenCL as follows;
|
|
|
|
```bash
|
|
sudo apt install libxcb-dri2-0 libxcb-dri3-0 libwayland-client0 libwayland-server0 libx11-xcb1
|
|
```
|
|
|
|
Now you can run "clinfo" to check whether OpenCL is working.
|
|
|
|
```bash
|
|
clinfo...
|
|
```
|
|
|
|
Check whether some dependencies are missing using ldd command as follows;
|
|
|
|
```
|
|
ldd /usr/lib/libmali-valhall-g610-g6p0-x11-wayland-gbm.so
|
|
```
|
|
|
|
Create Symbolic Link for libOpenCL.so
|
|
|
|
The directory "/usr/lib/aarch64-linux-gnu/" will have "libOpenCL.so.1.0.0". But no "libOpenCL.so" file. In this case, create a symbolic link as follows. You need to log into root account to create this, say, "su -";
|
|
|
|
```bash
|
|
cd /usr/lib/aarch64-linux-gnu/ && sudo ln -s libOpenCL.so.1.0.0 libOpenCL.so
|
|
```
|
|
|
|
Copying OpenCL "CL" Folder into "/usr/include"
|
|
|
|
```bash
|
|
cd ~/ && wget https://www.roselladb.com/download/CLv3.zip
|
|
unzip CLv3.zip
|
|
sudo cp -r CL /usr/include
|
|
sudo reboot
|
|
``` |