Barrelfish inside qemu with PCIe passthrough

Recent Intel/AMD processors and chipsets support PCIe virtualization. This requires a CPU supporting VT-d/AMD-Vi and a chipset that includes an IOMMU. Linux with KVM support passing a PCIe device to qemu guests transparently. This guide focuses on Intel hardware.

Prerequisites

A computer supporting IOMMU and a processor supporting VT-d are required for this to function. Additionally, a Linux version with KVM and IOMMU support has to be running. Check for IOMMU support using dmesg | grep IOMMU . The output should be similar to the following:

ubuntu@ubuntu:~$ dmesg|grep IOMMU
[    0.000000] Intel-IOMMU: enabled
[    0.000000] Intel-IOMMU: enabled
[    0.401164] dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0466 ecap f020de
[    0.416382] dmar: IOMMU 1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0466 ecap f020de
[    0.436580] IOAPIC id 2 under DRHD base  0xfbffe000 IOMMU 0
[    0.442900] IOAPIC id 0 under DRHD base  0xebffc000 IOMMU 1
[    0.449217] IOAPIC id 1 under DRHD base  0xebffc000 IOMMU 1
[   30.844389] IOMMU 0 0xfbffe000: using Queued invalidation
[   30.850527] IOMMU 1 0xebffc000: using Queued invalidation
[   30.856664] IOMMU: Setting RMRR:
[   30.860391] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbdcf9000 - 0xbdd1dfff]
[   30.869716] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbdcf9000 - 0xbdd1dfff]
[   30.879012] IOMMU: Prepare 0-16MiB unity mapping for LPC
[   30.885068] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

The IOMMU sometimes needs to be enabled explicitly from whithin the BIOS.

Preparations

This section assumes a Debian/Ubuntu host.

Setup kernel command line

Add intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 intel_iommu=on to the kernel command line. Maybe one also needs to add pci-stub.ids=VVVV:DDDD to the kernel parameter list with the correct vendor/device ID.

Bind device to vfio-pci

Install qemu-kvm. Create a script with the following contents. Here, we call it pci-setup.sh.

modprobe vfio-pci

for dev in "$@"; do
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
done

Determining device IDs

Use lspci -Dvvnn | grep $identifier to determine PCI bus/device/function numbers. The following examples shows the desired information for a Mellanox card:

ubuntu@ubuntu:~$ lspci -Dvvnn | grep Mellanox
0000:83:00.0 Network controller [0280]: Mellanox Technologies MT27500 Family [ConnectX-3] [15b3:1003]
        Subsystem: Mellanox Technologies Device [15b3:0050]

The device ID is 0000:83:00.0 in this case. Run the above script with the device ID as parameter. For example:

root@ubuntu:~# chmod u+x pci-setup.sh 
root@ubuntu:~# ./pci-setup.sh 0000:83:00.0

Passing the device to qemu

To run qemu with PCI forwarding enabled, add the following command line options: -enable-kvm -device vfio-pci,host=83:00.0, in this case we want to forward the device 83:00.0 to qemu.

Notes

One has to use vfio-pci. The old approach with pci-stub does not work.