I was curious if anyone else has seen this or had any suggestions.

I have recently began playing around with my two servers (Freenas Box and oVirt box) and their 10Gb Ethernet ports.

I can access the Freenas SMB share over the 10Gb port without issue and I have been playing around with the capabilities.  After finding out that my Linux Raid (MDADM) mirror is having horrible write performance, I decided to plug in an NVMe drive that I had lying around and check out its performance.

For my first test,I added the NVMe drive as a passthrough device to a Windows guest and was able to transfer to and from Freenas box without issue.  Speeds were typically ~350-400 MB/s but could drop down to 250 MB/s or so, and would top out around 525 MB/s, pretty slick!

For my second test, I decided to mount the NVMe drive on the CentOS ovirt host and make it a local datastore.  I migrated my Windows Guest to it, and decided to test and see what sort of transfer speeds I got and saw some weird results...

Writing TOO the NAS worked about the same.  Perhaps a little slower but at least had a steady 250-300 MB/s.

Writing to the Windows Guest had a very "Fast and then slow, fast and then slow" type of throughput.  I took a few screenshots:

(Writing TO the NAS was fairly consistent)

Inline image 1
https://i.imgur.com/jWNNvfp.png


(Writing TO the Windows Guest on NVMe storage)
Sometimes these hit the low 10-20 MB/s during the transfer.
Inline image 2
https://i.imgur.com/aizG6n0.png

Inline image 3
https://i.imgur.com/AjRpR0K.png