Hacker’s guide to the Raspberry Pi AI kit on Ubuntu

Raspberry Pi recently released their AI Kit, a PCIe M.2 HAT populated with a Hailo-8L AI accelerator. This AI accelerator has a performance of up to 13 tera-operations per second (TOPS). In comparison, the Coral Edge TPU is rated at 4 TOPS, which has a similar inference performance to the CPU of the Raspberry Pi 5. The AI Kit makes the development of machine learning workloads on a standard platform much easier. It also allows the deployment of a powerful Machine Learning model on the edge, at a low cost, and low power usage. Many are excited about the numerous use cases where this new kit could excel, such as real time object detection on camera feeds in security, manufacturing or quality assurance.

In this blog post, we will discuss our experiences trying to get the Hailo-8L accelerator and its software running on Ubuntu 24.04 LTS. The hardware and software from Hailo are still very new, so things still change often. The steps we use to get it working are not recommended for production use, but rather to explore the ecosystem and get a development setup going.

Hailo’s software stack

The Hailo accelerators do not directly support standard ML frameworks like TensorFlow. Therefore, you will need to use Hailo’s software stack. The software stack consists mainly of four parts:

• Hailo Driver: PCIe driver and firmware for Hailo-8L
• HailoRT: Hailo’s Runtime
• TAPPAS: Hailo’s development framework
• AI workload: Raspberry Pi 5 examples

Software for Raspberry Pi OS

Raspberry Pi has documented how to get the AI Kit to work, and how to run examples under Raspberry Pi OS. For the Raspberry Pi OS, the software components are packaged in debs, with the correct compatible versions available from Raspberry Pi’s Debian repository. 

One can also compile these software components from source, but due to the layers of dependencies, this is not an easy task. You also need to make sure to use compatible versions of all the different components. The sources are currently slightly newer than what is packaged for Raspberry Pi.

At the time of writing, the available versions are:

• Accelerator firmware:
  • Deb package: hailofw_4.17.0-2_all.deb
  • Source: download script for v4.18.0
• PCIe driver:
  • v4.17.0 included in Raspberry Pi kernel
  • Source code for v4.18.0
• Hailo Runtime:
  • Deb package: hailort_4.17.0_arm64.deb
  • Source code for v4.18.0
• TAPPAS:
  • Deb package: hailo-tappas-core-3.28.2_3.28.2_arm64.deb
  • Source code for v3.29.1
• Official Raspberry Pi examples:
  • Deb package: rpicam-apps-hailo-postprocess_1.5.0-2_arm64.deb
  • Source code
• Hailo’s examples for the Raspberry Pi 5:
  • Source code

Installing the software on Ubuntu 24.04

Our first approach is to install the Raspberry Pi deb packages on Ubuntu 24.04.

Driver and firmware

There is no Raspberry Pi debian package for the Hailo PCIe drivers. We therefore install it from source.

To compile the driver, you need the kernel headers installed on your system, as well as the standard build tools:

sudo apt install linux-headers-$(uname -r)
sudo apt install build-essential

Clone the v4.17.0 version of the source code using git. This is the version that is required by the rest of the software we will be installing from the deb packages.

git clone https://github.com/hailo-ai/hailort-drivers.git --branch v4.17.0
cd hailort-drivers/

The steps needed to compile the source code and then to install the driver, udev rules and firmware are well documented by Hailo. We recommend you create a Hailo account and read their guide.

Based on Hailo’s guide, do the following:

cd linux/pcie
make all
sudo make install
cd ../..
./download_firmware.sh
sudo mkdir -p /lib/firmware/hailo
sudo mv hailo8_fw.4.17.0.bin /lib/firmware/hailo/hailo8_fw.bin
sudo cp ./linux/pcie/51-hailo-udev.rules /etc/udev/rules.d/

Restart after installing the driver to make sure the firmware, udev rules and kernel module are all loaded correctly. If installation was successful, you should see a device under /dev/hailo* and some entries about it in dmesg. Make sure the driver version reported by dmesg is indeed v4.17.0.

$ ls /dev/hailo*
/dev/hailo0

$ sudo dmesg | grep hailo
[    4.980184] hailo_pci: loading out-of-tree module taints kernel.
[    4.980195] hailo_pci: module verification failed: signature and/or required key missing - tainting kernel
[    4.981731] hailo: Init module. driver version 4.17.0
[    4.981844] hailo 0000:01:00.0: Probing on: 1e60:2864...
[    4.981849] hailo 0000:01:00.0: Probing: Allocate memory for device extension, 11600
[    4.981859] hailo 0000:01:00.0: enabling device (0000 -> 0002)
[    4.981865] hailo 0000:01:00.0: Probing: Device enabled
[    4.981880] hailo 0000:01:00.0: Probing: mapped bar 0 - 00000000b70945f0 16384
[    4.981884] hailo 0000:01:00.0: Probing: mapped bar 2 - 00000000d21184c7 4096
[    4.981888] hailo 0000:01:00.0: Probing: mapped bar 4 - 00000000fde02b51 16384
[    4.981893] hailo 0000:01:00.0: Probing: Setting max_desc_page_size to 4096, (page_size=4096)
[    4.981901] hailo 0000:01:00.0: Probing: Enabled 64 bit dma
[    4.981906] hailo 0000:01:00.0: Probing: Using userspace allocated vdma buffers
[    4.981910] hailo 0000:01:00.0: Disabling ASPM L0s 
[    4.981914] hailo 0000:01:00.0: Successfully disabled ASPM L0s 
[    5.030118] UBSAN: array-index-out-of-bounds in /home/jpmeijers/hailort-drivers/linux/pcie/../../common/pcie_common.c:351:53
[    5.030152]  hailo_pcie_write_firmware+0x260/0x280 [hailo_pci]
[    5.030169]  hailo_pcie_probe+0x970/0xd98 [hailo_pci]
[    5.030204]  hailo_pcie_module_init+0x98/0xff8 [hailo_pci]
[    5.161028] hailo 0000:01:00.0: Firmware was loaded successfully
[    5.172591] hailo 0000:01:00.0: Probing: Added board 1e60-2864, /dev/hailo0

HialoRT

HailoRT – the Hailo Runtime is available as a Debian package from the Raspberry Pi repositories. Download the Debian package and then install it using apt:

wget http://archive.raspberrypi.com/debian/pool/main/h/hailort/hailort_4.17.0_arm64.deb
sudo apt install ./hailort_4.17.0_arm64.deb

After installing HailoRT, you should be able to communicate with the Hailo accelerator hardware. Check that communication is working by identifying connected devices:

$ hailortcli fw-control identify
Executing on device: 0000:01:00.0
Identifying board
Control Protocol Version: 2
Firmware Version: 4.17.0 (release,app,extended context switch buffer)
Logger Version: 0
Board Name: Hailo-8
Device Architecture: HAILO8L
Serial Number: HLDDLBB241602838
Part Number: HM21LB1C2LAE
Product Name: HAILO-8L AI ACC M.2 B+M KEY MODULE EXT TMP

TAPPAS

TAPPAS, the development framework for creating models to run on Hailo accelerators, is also available as a Debian package from the Raspberry Pi repositories. Download the Debian package and install it:

wget http://archive.raspberrypi.com/debian/pool/main/h/hailo-tappas-core-3.28.2/hailo-tappas-core-3.28.2_3.28.2_arm64.deb
sudo apt install ./hailo-tappas-core-3.28.2_3.28.2_arm64.deb

This may take a while as TAPPAS has a long list of dependencies that will be downloaded and installed by apt.

Official Raspberry Pi examples

Download location: rpicam-apps-hailo-postprocess_1.5.0-2_arm64.deb
It is not possible to install this package on Ubuntu, as it depends on rpicam-apps.

rpicam-apps-hailo-postprocess : Depends: rpicam-apps but it is not installable
E: Unable to correct problems, you have held broken packages.

The rpicam-apps package can also be downloaded from the Raspberry Pi archive, but trying to install it is impossible due to conflicts in dependencies.

 rpicam-apps : Depends: libavcodec59 (>= 7:5.0) but it is not installable
               Depends: libavdevice59 (>= 7:5.0) but it is not installable
               Depends: libavformat59 (>= 7:5.0) but it is not installable
               Depends: libavutil57 (>= 7:5.1) but it is not installable
               Depends: libcamera0.3 (>= 0.3.0+rpt20240617) but it is not installable
               Depends: libjpeg62-turbo (>= 1.3.1) but it is not installable

Hailo’s Raspberry Pi examples

Fortunately Hailo publishes examples for the Raspberry Pi 5 which do not depend on rpicam-apps. Hailo only publishes wheels for the hailo Python library for Python versions 3.8 – 3.11. Ubuntu 24.04 ships with Python 3.12. To solve this we install Python 3.11 from the deadsnakes ppa.

sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt install python3.11-full python3.11-dev

Then we change the system default Python to version 3.11:

sudo update-alternatives --install /usr/local/bin/python3 python3 /usr/bin/python3.11 3

Changing the system default Python version is required as the setup script for the examples always uses the system default Python version. Manually creating a virtual environment with Python 3.11, then running the setup script inside it does not work, as it yields an environment with Python 3.12, not Python 3.11 as is required.

Before you continue, make sure the Python interpreter that is available via the python3 command is indeed version 3.11:

$ python3 --version
Python 3.11.9

The compilation of the examples requires meson and ninja. When you install meson, it will automatically add ninja as a dependency:

sudo apt install meson

Then clone the examples repository, build it and install it, as documented here:

git clone https://github.com/hailo-ai/hailo-rpi5-examples.git
cd hailo-rpi5-examples
source setup_env.sh
pip install -r requirements.txt
./download_resources.sh
./compile_postprocess.sh

We can now run one of the examples:

(venv_hailo_rpi5_examples) jpmeijers@pi-2404-desktop:~/hailo-rpi5-examples$ python basic_pipelines/detection.py --input resources/detection0.mp4
Traceback (most recent call last):
  File "/home/jpmeijers/hailo-rpi5-examples/basic_pipelines/detection.py", line 1, in <module>
    import gi
  File "/usr/lib/python3/dist-packages/gi/__init__.py", line 40, in <module>
    from . import _gi
ImportError: cannot import name '_gi' from partially initialized module 'gi' (most likely due to a circular import) (/usr/lib/python3/dist-packages/gi/__init__.py)

To fix this error we need to re-install PyGObject inside the virtual environment, so that we have the correct version for Python 3.11. The error is due to Python 3.11 using the PyGObject library installed on the system, and which is meant for Python 3.12:

pip install --ignore-installed PyGObject

If we run the example again, we will hit one more error, this time about numpy. A simple upgrade to the latest numpy version should fix it:

pip install numpy --upgrade

Now we can run the examples.

python basic_pipelines/detection.py --input resources/detection0.mp4

Commands for other examples, or using a camera can be found in the examples’ documentation. We tested with a USB webcam, as the Raspberry Pi cameras aren’t well supported on Ubuntu yet. If you have a Raspberry Pi camera connected, along with a USB webcam, you might need to specify the correct input source. For example, our USB webcam is registered as /dev/video8:

python basic_pipelines/detection.py --input /dev/video8

If you opened a new terminal and want to run the examples there, you need to activate the virtual environment again.

cd hailo-rpi5-examples/
source setup_env.sh

Compiling from source

We tried compiling the software components from source. The PCIe driver and HailoRT successfully compiled, allowing us to communicate with the Hailo-8L accelerator from Ubuntu 24.04. We couldn’t compile TAPPAS due to several factors. This also meant we could not use the Raspberry Pi examples, as they depend on TAPPAS.

It is not impossible to get it to work, but manual changes to the source code are required. Some of the changes are documented by Matt Davis in his GitHub repository.

A better approach

Even though we got the Raspberry Pi 5 examples from Hailo to run on Ubuntu 24.04, it required replacing the default Python version. This is not recommended, as it may cause dependency or runtime failures at some later point in time. It is better to keep the host system on a stable set of packages and move the workload into an isolated environment.

We decided to mimic what Raspberry Pi OS does, by installing Debian Bookworm in a Docker container, followed by installing the Raspberry Pi deb packages for Hailo. Using this method, we still have the Hailo-8L hardware connected via the host operating system – Ubuntu 24.04. Fortunately, the PCIe driver works fine under Ubuntu, and we can access the device from the Docker container.

Another benefit of this is that any future updates to the Hailo Debian packages, which are targeted for Raspberry Pi OS, will have a higher chance of working in the container. This is because the same base operating system that is used for Raspberry Pi OS is used in the container.

Our example Dockerfile to do this can be seen here:

https://github.com/canonical/pi-ai-kit-ubuntu

If you follow the guide in this repository’s readme, you will notice that it requires a couple of manual steps which can not be performed during the Docker build process. Because of these manual steps, using the Docker container is also not suitable for production use cases.

Conclusion

It is possible to use the Raspberry Pi AI Kit on a Raspberry Pi 5, running Ubuntu 24.04 LTS. The process to get it to work is not user friendly and may break system compatibility. An alternative method of using a Docker container to isolate the AI workload from the host is also shown and recommended as an alternative. This comes with its own pitfalls, so it is also not suitable for a production environment.

This blog post provides examples of how the Raspberry Pi AI Kit can be used under Ubuntu 24.04. Due to the nature of this field, where software changes rapidly, we can not guarantee that this will keep on working. It is, however, a good starting point if you want to develop and test your AI applications on Ubuntu.

As the Hailo software stack matures, the method of using it will change. We hope to see a more production-ready solution from Hailo soon.