= Barrelfish inside qemu with PCIe passthrough =

Recent Intel/AMD processors and chipsets support PCIe virtualization. This requires a CPU supporting VT-d/AMD-Vi and a chipset that includes an IOMMU. Linux with KVM support passing a PCIe device to qemu guests transparently. This guide focuses on Intel hardware.

== Prerequisites ==

A computer supporting IOMMU and a processor supporting VT-d are required for this to function. Additionally, a Linux version with KVM and IOMMU support has to be running. Check for IOMMU support using {{{dmesg | grep IOMMU }}}. The output should be similar to the following:
{{{
ubuntu@ubuntu:~$ dmesg|grep IOMMU
[    0.000000] Intel-IOMMU: enabled
[    0.000000] Intel-IOMMU: enabled
[    0.401164] dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0466 ecap f020de
[    0.416382] dmar: IOMMU 1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0466 ecap f020de
[    0.436580] IOAPIC id 2 under DRHD base  0xfbffe000 IOMMU 0
[    0.442900] IOAPIC id 0 under DRHD base  0xebffc000 IOMMU 1
[    0.449217] IOAPIC id 1 under DRHD base  0xebffc000 IOMMU 1
[   30.844389] IOMMU 0 0xfbffe000: using Queued invalidation
[   30.850527] IOMMU 1 0xebffc000: using Queued invalidation
[   30.856664] IOMMU: Setting RMRR:
[   30.860391] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbdcf9000 - 0xbdd1dfff]
[   30.869716] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbdcf9000 - 0xbdd1dfff]
[   30.879012] IOMMU: Prepare 0-16MiB unity mapping for LPC
[   30.885068] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
}}}

The IOMMU sometimes needs to be enabled explicitly from whithin the BIOS.

== Preparations ==

This section assumes a Debian/Ubuntu host. 

=== Setup kernel command line ===

Add {{{intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 intel_iommu=on}}} to the kernel command line. Maybe one also needs to add {{{pci-stub.ids=VVVV:DDDD}}} to the kernel parameter list with the correct vendor/device ID.

=== Bind device to vfio-pci ===

Install {{{qemu-kvm}}}. Create a script with the following contents. Here, we call it pci-setup.sh.
{{{
#!/bin/bash

modprobe vfio-pci

for dev in "$@"; do
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
done
}}}

=== Determining device IDs ===

Use {{{lspci -Dvvnn | grep $identifier}}} to determine PCI bus/device/function numbers. The following examples shows the desired information for a Mellanox card:
{{{
ubuntu@ubuntu:~$ lspci -Dvvnn | grep Mellanox
0000:83:00.0 Network controller [0280]: Mellanox Technologies MT27500 Family [ConnectX-3] [15b3:1003]
        Subsystem: Mellanox Technologies Device [15b3:0050]
}}}
The device ID is {{{0000:83:00.0}}} in this case. Run the above script with the device ID as parameter. For example: {{{
root@ubuntu:~# chmod u+x pci-setup.sh 
root@ubuntu:~# ./pci-setup.sh 0000:83:00.0
}}}

=== Passing the device to qemu ===

To run qemu with PCI forwarding enabled, add the following command line options: {{{-enable-kvm -device vfio-pci,host=83:00.0}}}, in this case we want to forward the device {{{83:00.0}}} to qemu.

== Notes ==

One has to use vfio-pci. The old approach with pci-stub does not work.

== Resources ==

 * [[https://bbs.archlinux.org/viewtopic.php?id=162768&p=60]]
 * [[https://wiki.debian.org/VGAPassthrough]]