Ubuntu errors with PCIPassthrough with GPUs

edit

I have an OS Mitaka deployment that was done by Fuel (9.0).

I have a system with 8GPUs in a single box. We are trying to allow VMs to request access to GPU resources via this box.

I know that with PCI Passthrough you can't share a device with a VM (e.g. deviceid1 <-> 1 VM, or deviceid2 & deviceid3 <-> 1VM).

I have successfully been able to get the system to have 1 GPU <-> 1 VM with the CentOS image from here (http://docs.openstack.org/image-guide...). However, if I try to use the Ubuntu 14.04 image it doesn't attach properly.

I see the following in dmesg. [ 0.638033] pci 0000:00:05.0: unknown header type 7f, ignoring device

At that point the device on the host system goes

from

0d:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX TITAN X] [10de:17c2] (rev a1) (prog-if 00 [VGA controller])

to

0d:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX TITAN X] [10de:17c2] (rev ff) (prog-if ff)

At this point the device is "busy" and never released properly. I'm not sure if this is an "image" problem, or a KVM/qemu/libvirt issue.

A few questions.

1. Does the Ubuntu image listed on the Openstack site support GPUs?
2. If not how do I add the option to the Ubuntu image to accept GPU devices?
3. How do I enable an image (centOS or Ubuntu) to accept more than 1 GPU attached to a single VM? (I know the same GPU can't be assigned to multiple VMs).

I was able to reproduce this same issue with CentOS by trying to assign 2x GPUs to a single VM, 1 GPU attached and the other went into the (ff) (prog-if ff) state.