NVIDIA
List of GPU's that Support DDA in Server 2016 for GPU Pass Through in Hyper-V VM's
I'm going to lead off with the fact that I do not represent a corporate entity which means I don't have the means to purchase GRID or Tesla cards or licensing specifically for this purpose. This is purely for educational, proof of concept & testing purposes in a home lab environment. As the topic suggests, I'm just looking for a list of non-GRID non-Tesla GPUs that support Discrete Device Assignment (DDA) in Server 2016 so I can pass through GPU's (or vGPU's if the card supports it) to a Hyper-V VM. My objective is to [olist] [.]locate some sort of datasheet with an exhaustive list[/.] [.]understand whether or not it's possible to know which cards do support DDA[/.] [.]understand which cards (outside of high-end cards like GRID and Tesla) support vGPU's[/.] [.]check for any get firsthand accounts from others who have done this complete with gotcha's, lessons learned etc[/.] [/olist] I've seem some reports that suggest it [i]may[/i] be possible to get consumer grade cards (e.g.: GeForce) working in VM's. However, the processes are not well documented and the explanations seemed really convoluted, so I don't want to waste cycles jumping through a bunch of hoops. Plus I don't want the host BSOD'ing periodically either because I'm doing something super hackish. I've read some things online that suggest [i]any[/i] Quardo card (e.g.: Quadro NVS, Quadro FX etc.) [i]should[/i] support DDA/pass through. However I can't find any literature to support those claims, and all the examples I've come across are K1's, Teslas etc. Since I don't have the means to burn through various models (e.g.: FX 3800, NVS 420, 600 etc) in trial & error, I'm reaching out to the community for guidance.
I'm going to lead off with the fact that I do not represent a corporate entity which means I don't have the means to purchase GRID or Tesla cards or licensing specifically for this purpose. This is purely for educational, proof of concept & testing purposes in a home lab environment.

As the topic suggests, I'm just looking for a list of non-GRID non-Tesla GPUs that support Discrete Device Assignment (DDA) in Server 2016 so I can pass through GPU's (or vGPU's if the card supports it) to a Hyper-V VM.

My objective is to
  1. locate some sort of datasheet with an exhaustive list
  2. understand whether or not it's possible to know which cards do support DDA
  3. understand which cards (outside of high-end cards like GRID and Tesla) support vGPU's
  4. check for any get firsthand accounts from others who have done this complete with gotcha's, lessons learned etc


I've seem some reports that suggest it may be possible to get consumer grade cards (e.g.: GeForce) working in VM's. However, the processes are not well documented and the explanations seemed really convoluted, so I don't want to waste cycles jumping through a bunch of hoops. Plus I don't want the host BSOD'ing periodically either because I'm doing something super hackish.

I've read some things online that suggest any Quardo card (e.g.: Quadro NVS, Quadro FX etc.) should support DDA/pass through. However I can't find any literature to support those claims, and all the examples I've come across are K1's, Teslas etc. Since I don't have the means to burn through various models (e.g.: FX 3800, NVS 420, 600 etc) in trial & error, I'm reaching out to the community for guidance.

#1
Posted 06/29/2018 08:00 PM   
[quote=""]understand whether or not it's possible to know which cards do support DDA[/quote] NVidia does not update list of this "Multi-OS" technology (see [url]http://www.nvidia.com/object/sli_multi_os.html[/url]) and NVidia does not answer any question or test or support this. GPU passthrough should be supported for [b]Quadro >= x2000[/b] (or Tesla/Grid) cards. This means that guest driver does not refuse to accept card (eg. not need to mask hypervisor presence (search for KVM GPU stories) or modify guest driver binaries). But the correct behavior depends on hardware, BIOS/UEFI and hypervisor support (like CPU+chipset, IOMMU (VT-d/AMD-Vi), MSI/MSI-X and IRQ remapping, support for 64 PCI BAR ...). Only few combinations are "certified/working" in commercial hypervisors (check - [url]http://hcl.xenserver.org/gpus/?features=1[/url] or [url]https://www.vmware.com/resources/compatibility/search.php?deviceCategory=vdga[/url] ... but no matrix for DDA). You can read my story with [b]Quadro P2000[/b] - [url]https://devtalk.nvidia.com/default/topic/992447/[/url]. [quote=""]understand which cards (outside of high-end cards like GRID and Tesla) support vGPU's[/quote] Officially none but it is possible with linux based hypervisors (see [url]https://gridforums.nvidia.com/default/topic/8934/[/url]).
said:understand whether or not it's possible to know which cards do support DDA

NVidia does not update list of this "Multi-OS" technology (see http://www.nvidia.com/object/sli_multi_os.html) and NVidia does not answer any question or test or support this. GPU passthrough should be supported for Quadro >= x2000 (or Tesla/Grid) cards. This means that guest driver does not refuse to accept card (eg. not need to mask hypervisor presence (search for KVM GPU stories) or modify guest driver binaries). But the correct behavior depends on hardware, BIOS/UEFI and hypervisor support (like CPU+chipset, IOMMU (VT-d/AMD-Vi), MSI/MSI-X and IRQ remapping, support for 64 PCI BAR ...). Only few combinations are "certified/working" in commercial hypervisors (check - http://hcl.xenserver.org/gpus/?features=1 or https://www.vmware.com/resources/compatibility/search.php?deviceCategory=vdga ... but no matrix for DDA). You can read my story with Quadro P2000 - https://devtalk.nvidia.com/default/topic/992447/.

said:understand which cards (outside of high-end cards like GRID and Tesla) support vGPU's

Officially none but it is possible with linux based hypervisors (see https://gridforums.nvidia.com/default/topic/8934/).

#2
Posted 06/29/2018 09:02 PM   
[quote="mcerveny"]NVidia does not update list of this "Multi-OS" technology (see [url]http://www.nvidia.com/object/sli_multi_os.html[/url]) ... GPU passthrough should be supported for [b]Quadro >= x2000[/b] (or Tesla/Grid) cards. This means that guest driver does not refuse to accept card (eg. not need to mask hypervisor presence (search for KVM GPU stories) or modify guest driver binaries).[/quote] Bummer they don't maintain that list, but what's there is helpful - thank you! When you say Quadro >= x2000 am I correct in understanding that this means Quadro cards numbering 2000 and up such as Quadro 2200D, Quadro FX 3800, Quadro K2200 but [b][i]not[/i][/b] something like a Quadro NVS 420, right? [quote="mcerveny"]But the correct behavior depends on hardware, BIOS/UEFI and hypervisor support (like CPU+chipset, IOMMU (VT-d/AMD-Vi), MSI/MSI-X and IRQ remapping, support for 64 PCI BAR ...).[/quote] Understood - I failed to mention I'm doing this on a Dell PowerEdge R710 Gen II. [quote="mcerveny"]Only few combinations are "certified/working" in commercial hypervisors (check - [url]http://hcl.xenserver.org/gpus/?features=1[/url] or [url]https://www.vmware.com/resources/compatibility/search.php?deviceCategory=vdga[/url] ... but no matrix for DDA).[/quote] I find that XenServer site very useful - thank you again. [quote="mcerveny"]You can read my story with [b]Quadro P2000[/b] - [url]https://devtalk.nvidia.com/default/topic/992447/[/url]. Officially none but it is possible (see [url]https://gridforums.nvidia.com/default/topic/8934/[/url]). [/quote] I'm intellectually interested in this, but a little concerned it's a tad over my head. :) Also Windows Server 2016 would be the host so I'm unsure if it would translate the same way. I'm open to trying it out but I would need some help getting caught up to your level of intelligence on this subject.
mcerveny said:NVidia does not update list of this "Multi-OS" technology (see http://www.nvidia.com/object/sli_multi_os.html) ... GPU passthrough should be supported for Quadro >= x2000 (or Tesla/Grid) cards. This means that guest driver does not refuse to accept card (eg. not need to mask hypervisor presence (search for KVM GPU stories) or modify guest driver binaries).

Bummer they don't maintain that list, but what's there is helpful - thank you! When you say Quadro >= x2000 am I correct in understanding that this means Quadro cards numbering 2000 and up such as Quadro 2200D, Quadro FX 3800, Quadro K2200 but not something like a Quadro NVS 420, right?


mcerveny said:But the correct behavior depends on hardware, BIOS/UEFI and hypervisor support (like CPU+chipset, IOMMU (VT-d/AMD-Vi), MSI/MSI-X and IRQ remapping, support for 64 PCI BAR ...).

Understood - I failed to mention I'm doing this on a Dell PowerEdge R710 Gen II.

mcerveny said:Only few combinations are "certified/working" in commercial hypervisors (check - http://hcl.xenserver.org/gpus/?features=1 or https://www.vmware.com/resources/compatibility/search.php?deviceCategory=vdga ... but no matrix for DDA).

I find that XenServer site very useful - thank you again.

mcerveny said:You can read my story with Quadro P2000 - https://devtalk.nvidia.com/default/topic/992447/. Officially none but it is possible (see https://gridforums.nvidia.com/default/topic/8934/).

I'm intellectually interested in this, but a little concerned it's a tad over my head. :) Also Windows Server 2016 would be the host so I'm unsure if it would translate the same way. I'm open to trying it out but I would need some help getting caught up to your level of intelligence on this subject.

#3
Posted 06/30/2018 02:00 PM   
Hi Julius, Your host won't even support DDA at all. You should start to check your hardware first before starting with anything else. Here a script to check for DDA requirements as starting point: https://raw.githubusercontent.com/Microsoft/Virtualization-Documentation/master/hyperv-samples/benarm-powershell/DDA/survey-dda.ps1 Regards Simon
Hi Julius,

Your host won't even support DDA at all. You should start to check your hardware first before starting with anything else.

Here a script to check for DDA requirements as starting point:

https://raw.githubusercontent.com/Microsoft/Virtualization-Documentation/master/hyperv-samples/benarm-powershell/DDA/survey-dda.ps1


Regards

Simon

#4
Posted 07/01/2018 07:29 AM   
https://www.youtube.com/watch?v=k1oHCjEQsnc [i]If Vmware vDGA works (all HW/BIOS/vSphere hypervisor is OK) then Hyper-V DDA should work too.[/i]


If Vmware vDGA works (all HW/BIOS/vSphere hypervisor is OK) then Hyper-V DDA should work too.

#5
Posted 07/01/2018 05:16 PM   
Martin, this is not correct. DDA requires SR-IOV support. But you are right with R710 Revision II. I just checked and for Revision II Dell provided a BIOS update to support SR-IOV. R710 (Revision I) does not support DDA! regards Simon
Martin,

this is not correct. DDA requires SR-IOV support. But you are right with R710 Revision II. I just checked and for Revision II Dell provided a BIOS update to support SR-IOV. R710 (Revision I) does not support DDA!

regards
Simon

#6
Posted 07/02/2018 07:21 AM   
Hi & Good morning everyone First, thanks for taking the time to reply to this; I really do appreciate the feedback. [quote="sschaber"]Your host won't even support DDA at all. You should start to check your hardware first before starting with anything else. ... this is not correct. DDA requires SR-IOV support. But you are right with R710 Revision II. I just checked and for Revision II Dell provided a BIOS update to support SR-IOV. R710 (Revision I) does not support DDA![/quote] Correct, R710 Gen II or Revsion II does support SR-IOV provided it's running the proper BIOS version. For anyone else who stumbles across this post, additional information on that can be found in the links below (among others): [url]http://en.community.dell.com/techcenter/os-applications/w/wiki/3921.enabling-sr-iov-technology-on-delltm-poweredgetm-servers[/url] [url]http://en.community.dell.com/techcenter/os-applications/w/wiki/3459.dell-supported-platforms-for-windows-server-2012-sr-iov-feature[/url] FYI Simon thanks for posting that script; I'd forgotten about it. More details on that can be found here: [url]https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/plan/plan-for-deploying-devices-using-discrete-device-assignment[/url]
Hi & Good morning everyone
First, thanks for taking the time to reply to this; I really do appreciate the feedback.

sschaber said:Your host won't even support DDA at all. You should start to check your hardware first before starting with anything else.
...
this is not correct. DDA requires SR-IOV support. But you are right with R710 Revision II. I just checked and for Revision II Dell provided a BIOS update to support SR-IOV. R710 (Revision I) does not support DDA!


Correct, R710 Gen II or Revsion II does support SR-IOV provided it's running the proper BIOS version. For anyone else who stumbles across this post, additional information on that can be found in the links below (among others):
http://en.community.dell.com/techcenter/os-applications/w/wiki/3921.enabling-sr-iov-technology-on-delltm-poweredgetm-servers
http://en.community.dell.com/techcenter/os-applications/w/wiki/3459.dell-supported-platforms-for-windows-server-2012-sr-iov-feature

FYI Simon thanks for posting that script; I'd forgotten about it. More details on that can be found here:
https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/plan/plan-for-deploying-devices-using-discrete-device-assignment

#7
Posted 07/02/2018 02:30 PM   
Scroll To Top

Add Reply