NVIDIA
testing with GPU in passthrough
I have K2 in pass through mode on xenserver 6.2 SP1 and am trying to determine how my applications are stressing the GPU. I added the nvidia WMI pack, but windows perf monitor still shows 0 for all the GPU values I care about, such as memory and % GPU.
I have K2 in pass through mode on xenserver 6.2 SP1 and am trying to determine how my applications are stressing the GPU. I added the nvidia WMI pack, but windows perf monitor still shows 0 for all the GPU values I care about, such as memory and % GPU.

#1
Posted 09/29/2014 11:48 PM   
I forgot to add that nvidia-smi on the windows Vm does not show much data either, because WDDM is prohibitnig it from getting GPU memory usage. GPU usage shows 0%, which I doubt, when i have 12 users logged in with a NASA worldwind client.
I forgot to add that nvidia-smi on the windows Vm does not show much data either, because WDDM is prohibitnig it from getting GPU memory usage. GPU usage shows 0%, which I doubt, when i have 12 users logged in with a NASA worldwind client.

#2
Posted 09/30/2014 12:00 AM   
Hi William, Can you confirm that you're not using the driver for vGPU. If you are, can you switch to the passthrough driver from here. http://www.nvidia.co.uk/download/driverResults.aspx/78323/en-uk
Hi William,

Can you confirm that you're not using the driver for vGPU.

If you are, can you switch to the passthrough driver from here.


http://www.nvidia.co.uk/download/driverResults.aspx/78323/en-uk

Jason Southern, Regional Lead for ProVis Sales - EMEA: NVIDIA Ltd.

#3
Posted 09/30/2014 11:09 AM   
Yes, I am using the vGPU driver 332.83. I need vGPU to slice the GPU up with Xenapp to distribute the application to as many users as possible.
Yes, I am using the vGPU driver 332.83. I need vGPU to slice the GPU up with Xenapp to distribute the application to as many users as possible.

#4
Posted 09/30/2014 03:42 PM   
William, I am unclear on how you have this configured. If you are using pass-thru then use the normal NVIDIA driver for your OS (Jason provided a link above), this pins the GPU to that guest running XenApp and gives it the full performance of that GPU. Or are you using vGPU on XenServer to share out the GPUs? To be clear, vGPU is not the same as pass-thru, and I don't believe is supported with XenApp. Regardless, double sharing (vGPU shares the GPU first, then XenApp shares it again) does not make a ton of sense from a performance stand point. If you could provide more detail we can dig into this further for you.
William, I am unclear on how you have this configured. If you are using pass-thru then use the normal NVIDIA driver for your OS (Jason provided a link above), this pins the GPU to that guest running XenApp and gives it the full performance of that GPU. Or are you using vGPU on XenServer to share out the GPUs? To be clear, vGPU is not the same as pass-thru, and I don't believe is supported with XenApp. Regardless, double sharing (vGPU shares the GPU first, then XenApp shares it again) does not make a ton of sense from a performance stand point. If you could provide more detail we can dig into this further for you.

Regards,

Luke Wignall
Performance Engineering Manager
NVIDIA | Worldwide Sales ­ GRID Computing
http://www.linkedin.com/in/lukewignall/
https://twitter.com/lwignall

#5
Posted 10/01/2014 02:38 AM   
OK, I removed the vGPU driver and installed 332.84 version for windows 2008R2. (the 340.84 version would not extract). I stall cannot see any GPU utilization with nvidia-smi.exe or windows performance monitor.
OK, I removed the vGPU driver and installed 332.84 version for windows 2008R2. (the 340.84 version would not extract). I stall cannot see any GPU utilization with nvidia-smi.exe or windows performance monitor.

#6
Posted 10/03/2014 12:18 AM   
Are you using RDP to connect to the guest?
Are you using RDP to connect to the guest?

Regards,

Luke Wignall
Performance Engineering Manager
NVIDIA | Worldwide Sales ­ GRID Computing
http://www.linkedin.com/in/lukewignall/
https://twitter.com/lwignall

#7
Posted 10/03/2014 01:29 AM   
No, I am useing CITRIX ICA via xenapp. Nvidia-smi shows the PID for the applications I am using, but memory usage as insuffieicnt permissions and a value of NA. Is that because of the WDDM driver in use? I am testing both K1 and K2 cards and cannot make a determination of which I should use, since I dont have any data to base it on. My NASA worldwind looks good for 20 users, but I need data to prove scalability and choice of video cards. thanks, Willie Harrington
No, I am useing CITRIX ICA via xenapp. Nvidia-smi shows the PID for the applications I am using, but memory usage as insuffieicnt permissions and a value of NA. Is that because of the WDDM driver in use? I am testing both K1 and K2 cards and cannot make a determination of which I should use, since I dont have any data to base it on. My NASA worldwind looks good for 20 users, but I need data to prove scalability and choice of video cards. thanks, Willie Harrington

#8
Posted 10/03/2014 03:09 PM   
Hi William, You can certainly measure GPU information from within an ICA session. You must measure it at the server console. Since this is a VM, and since XenCenter doesn't allow you to connect a console session to a VM with a GPU in passthrough you either need to: I've sucessfully tested the following methods for monitoring the server GPU resources. a. Using Perfmon in an ICA session b. Use VNC to make a console connection c. Use RDP to make a console / admin connection (note that you can monitor the metrics but can't run GPU applications through RDP as Windows disables hardware acceleration for RDP sessions) d. Connecting from a remote machine in the domain. This was tested on a Server 2012R2 machine running XenApp 7.5 Remember that you're measuring the resources for the whole server, not for the session as there are no session metrics reported.
Hi William,


You can certainly measure GPU information from within an ICA session.

You must measure it at the server console. Since this is a VM, and since XenCenter doesn't allow you to connect a console session to a VM with a GPU in passthrough you either need to:

I've sucessfully tested the following methods for monitoring the server GPU resources.

a. Using Perfmon in an ICA session
b. Use VNC to make a console connection
c. Use RDP to make a console / admin connection (note that you can monitor the metrics but can't run GPU applications through RDP as Windows disables hardware acceleration for RDP sessions)
d. Connecting from a remote machine in the domain.

This was tested on a Server 2012R2 machine running XenApp 7.5

Remember that you're measuring the resources for the whole server, not for the session as there are no session metrics reported.

Jason Southern, Regional Lead for ProVis Sales - EMEA: NVIDIA Ltd.

#9
Posted 10/05/2014 06:58 PM   
To add to my note above, I think that the root cause of your issue is the original use of the vGPU driver. If you remove this and the WMI pack completely, then start again with the pass-through driver you should see the metrics start reporting. This links to a quick video of it in action, reporting within the ICA session https://onedrive.live.com/redir?resid=399BC7F1B763D265!3487&authkey=!ANU_m_vj2KMZZ-Y&ithint=video%2cmov I've an extended version which I'll post to YouTube once it's cleaned up, so you can see that it is possible to monitor within an ICA session, if you use the right driver.
To add to my note above, I think that the root cause of your issue is the original use of the vGPU driver.

If you remove this and the WMI pack completely, then start again with the pass-through driver you should see the metrics start reporting.

This links to a quick video of it in action, reporting within the ICA session


https://onedrive.live.com/redir?resid=399BC7F1B763D265!3487&authkey=!ANU_m_vj2KMZZ-Y&ithint=video%2cmov


I've an extended version which I'll post to YouTube once it's cleaned up, so you can see that it is possible to monitor within an ICA session, if you use the right driver.

Jason Southern, Regional Lead for ProVis Sales - EMEA: NVIDIA Ltd.

#10
Posted 10/06/2014 08:43 AM   
When I removed the WMI pack, I dont see any options under performance monitor for Nvidia any longer. I did get GPU-Z to give me some data, attached. What is the max I should use for memory on a K2 card in pass through? I dont have URL to post the pic, but I show a max of 31% for GPU load and 1047 MD dedicated memoyr and 431 usage (dynamic) for 16 users, each playing a mpg4 video, and NASA worldwind. [url][/url]
When I removed the WMI pack, I dont see any options under performance monitor for Nvidia any longer. I did get GPU-Z to give me some data, attached. What is the max I should use for memory on a K2 card in pass through? I dont have URL to post the pic, but I show a max of 31% for GPU load and 1047 MD dedicated memoyr and 431 usage (dynamic) for 16 users, each playing a mpg4 video, and NASA worldwind.
[url][/url]

#11
Posted 10/06/2014 08:49 PM   
[quote="william"]When I removed the WMI pack, I dont see any options under performance monitor for Nvidia any longer[/quote] Did you remove the drivers as well, reboot then install the 340 drivers? These are the drivers for Server 2008R2 and Server 2012R2, install these. http://www.nvidia.com/download/driverResults.aspx/78297/en-us [quote="william"]What is the max I should use for memory on a K2 card in pass through? [/quote] Not sure I understand this question. You don't allocate the GPU memory in passthrough it's fixed at 4GB per GPU (so 8 GB per Card). In XenApp the sharing is handled by XenApp and the Server OS, you have no control over it and to determine capacity you would need to load test as with any XenApp workload as load doesn't increase utilisation linearly.
william said:When I removed the WMI pack, I dont see any options under performance monitor for Nvidia any longer


Did you remove the drivers as well, reboot then install the 340 drivers?

These are the drivers for Server 2008R2 and Server 2012R2, install these.
http://www.nvidia.com/download/driverResults.aspx/78297/en-us


william said:What is the max I should use for memory on a K2 card in pass through?


Not sure I understand this question. You don't allocate the GPU memory in passthrough it's fixed at 4GB per GPU (so 8 GB per Card).

In XenApp the sharing is handled by XenApp and the Server OS, you have no control over it and to determine capacity you would need to load test as with any XenApp workload as load doesn't increase utilisation linearly.

Jason Southern, Regional Lead for ProVis Sales - EMEA: NVIDIA Ltd.

#12
Posted 10/07/2014 12:47 PM   
OK, after doing it in the correct order, I get the NVIDIA data in performance manager. Thanks much. Now, how to interpret what I am seeing. With 18 users, I am gettng 61% GPU utilization and 1388 MB memory allocation (I assume out of 4 GB) and 584 dynamic memory. So am I nearing the max I can expect from this one GPU? I also notice that CPU 0 and 2 are maxed at 100%, but I guess I will have to take that up with CITRIX and Microsoft forums.
OK, after doing it in the correct order, I get the NVIDIA data in performance manager. Thanks much.
Now, how to interpret what I am seeing. With 18 users, I am gettng 61% GPU utilization and 1388 MB memory allocation (I assume out of 4 GB) and 584 dynamic memory. So am I nearing the max I can expect from this one GPU? I also notice that CPU 0 and 2 are maxed at 100%, but I guess I will have to take that up with CITRIX and Microsoft forums.

#13
Posted 10/07/2014 10:28 PM   
Based on the numbers you could probably get an additional 8-10 users on pushing you into 90% utilisation territory. The likely reason for the CPU's being maxed at 100% is ctxgfx.exe. This is the HDX encoder process which is busy rendering all 18 of those sessions a 30fps (or whatever it can achieve based on resolution, CPU speed, available resource etc). The video below shows just this behaviour. Although in the video I'm using YouTube in Chrome, the effect on the CPU is the same. https://www.youtube.com/watch?v=GAHv_YcJsQ8
Based on the numbers you could probably get an additional 8-10 users on pushing you into 90% utilisation territory.

The likely reason for the CPU's being maxed at 100% is ctxgfx.exe. This is the HDX encoder process which is busy rendering all 18 of those sessions a 30fps (or whatever it can achieve based on resolution, CPU speed, available resource etc).

The video below shows just this behaviour. Although in the video I'm using YouTube in Chrome, the effect on the CPU is the same.

Jason Southern, Regional Lead for ProVis Sales - EMEA: NVIDIA Ltd.

#14
Posted 10/09/2014 10:55 AM   
How did you get such good video quality with Xenapp? without lossless HDX? My testing showed very fuzzy videos, with my video player or Windows media player with .mpg4 files. I had plenty of CPU resources remaining. I am using Open Gl exclusively. I retested on XenDesktop 7.1 and the HDX made all the difference in the world. Pretty near native video player performance.
How did you get such good video quality with Xenapp? without lossless HDX? My testing showed very fuzzy videos, with my video player or Windows media player with .mpg4 files. I had plenty of CPU resources remaining. I am using Open Gl exclusively.
I retested on XenDesktop 7.1 and the HDX made all the difference in the world. Pretty near native video player performance.

#15
Posted 10/14/2014 12:05 PM   
Scroll To Top

Add Reply