[Users] VirtIO disk latency

This is a multi-part message in MIME format. ------=_NextPartTM-000-e5c11ff1-4bb3-4471-9f08-8adebae2667b Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hello,=0A= =0A= coming from the "low cost NFS storage thread" I will open a new one=0A= about a topic that might be interesting for others too.=0A= =0A= We see a quite a heavy latency penalty using KVM VirtIO disks in comparison= =0A= to ESX. Doing one I/O onto disk inside a VM usually adds 370us of overhead = in=0A= the virtualisation layer. This has been tested with VirtIO-SCSI and windows= =0A= guest (2K3). More here (still now answer yet):=0A= =0A= http://lists.nongnu.org/archive/html/qemu-discuss/2013-12/msg00028.html=0A= =0A= A comparison for small sequential 1K I/Os on a NFS datastore in our setup g= ives:=0A= =0A= - access NFS inside the hypervisor - 12.000 I/Os per second - or 83us laten= cy =0A= - access DISK inside ESX VM that resides on NFS - 8000 I/Os per second - or= 125us latency=0A= - access DISK inside OVirt VM that resides on NFS - 2200 I/Os per second - = or 450us latency=0A= =0A= Even the official document at http://www.linux-kvm.org/page/Virtio/Block/La= tency=0A= suggest that the several mechanisms (iothread/vcpu) at least have a overhea= d=0A= of more than 200us=0A= =0A= Has anyone experienced something simlar. If these latency are normal it wou= ld =0A= make no sense to think about SSDs inside a central storage (be it ISCSI or = NFS =0A= or whatever).=0A= =0A= Markus=0A= =0A= ------=_NextPartTM-000-e5c11ff1-4bb3-4471-9f08-8adebae2667b Content-Type: text/plain; name="InterScan_Disclaimer.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="InterScan_Disclaimer.txt" **************************************************************************** Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. Über das Internet versandte E-Mails können unter fremden Namen erstellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserklärung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln Vorstand: Kadir Akin Dr. Michael Höhnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht Köln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln executive board: Kadir Akin Dr. Michael Höhnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497 **************************************************************************** ------=_NextPartTM-000-e5c11ff1-4bb3-4471-9f08-8adebae2667b--

On Thu, Jan 9, 2014 at 10:16 AM, Markus Stockhausen <stockhausen@collogia.de> wrote: ...
- access NFS inside the hypervisor - 12.000 I/Os per second - or 83us latency - access DISK inside ESX VM that resides on NFS - 8000 I/Os per second - or 125us latency - access DISK inside OVirt VM that resides on NFS - 2200 I/Os per second - or 450us latency
I can do a bit of testing on local disk and FC (with some extra setup maybe also NFS). What is your exact testing method? ( commands, file sizes, sofware versions, mount options etc.)

This is a multi-part message in MIME format. ------=_NextPartTM-000-269ed284-0ba0-45cc-bfe3-0742919cf61b Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable > Von: sander.grendelman@gmail.com =0A= > Gesendet: Donnerstag, 9. Januar 2014 10:32=0A= > An: Markus Stockhausen=0A= > Cc: users@ovirt.org=0A= > Betreff: Re: [Users] VirtIO disk latency=0A= > =0A= > On Thu, Jan 9, 2014 at 10:16 AM, Markus Stockhausen=0A= > <stockhausen@collogia.de> wrote:=0A= > ...=0A= > > - access NFS inside the hypervisor - 12.000 I/Os per second - or 83us l= atency=0A= > > - access DISK inside ESX VM that resides on NFS - 8000 I/Os per second = - or 125us latency=0A= > > - access DISK inside OVirt VM that resides on NFS - 2200 I/Os per secon= d - or 450us latency=0A= > =0A= > I can do a bit of testing on local disk and FC (with some extra setup=0A= > maybe also NFS).=0A= > What is your exact testing method? ( commands, file sizes, sofware=0A= > versions, mount options etc.)=0A= =0A= Thanks for taking time to help. =0A= =0A= I have used several tools to measure latencies but it=0A= always boils down to the same numbers. The software =0A= components and their releases should not matter to get =0A= a first overview. The important thing is to ensure that a =0A= read request of a test inside the VM is really passing the =0A= QEMU layer. =0A= =0A= The simplest test I can think of (at least in our case) is to=0A= start a Windows VM and attach a very small NFS disk with=0A= 1GB to it. Start it, install HDTune Trial and run the random =0A= access test to the small disk. Other ways could be to run =0A= some kind of direct IO based read test inside the VM.=0A= =0A= During each test I can see the packets running between =0A= the NFS server and the hypervisor so I know that each=0A= request is not cached inside the VM or QEMU. =0A= =0A= After one or two runs the filecache in the RAM of our NFS =0A= server has all the hot data and latency decreases down to=0A= the microseconds area. With that we can derive the=0A= penalty of the virtualization layer.=0A= =0A= Whatever I try to optimize I only reach 1/4th of the I/Os=0A= of ESX for very small packets (512 bytes or 1K). And=0A= that inside the same (migrated) VM on the same NFS=0A= topology with the same test programs.=0A= =0A= The baseline numbers for the hypervisor are an average of =0A= running direct io based test tools onto files residing on=0A= the same NFS.=0A= =0A= Markus=0A= =0A= P.S. I'm not complaining about that performance. =0A= Driving an IPoIB environment you get used to waste =0A= bandwidth and latency. But it is always good to know=0A= where it comes from.= ------=_NextPartTM-000-269ed284-0ba0-45cc-bfe3-0742919cf61b Content-Type: text/plain; name="InterScan_Disclaimer.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="InterScan_Disclaimer.txt" **************************************************************************** Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. Über das Internet versandte E-Mails können unter fremden Namen erstellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserklärung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln Vorstand: Kadir Akin Dr. Michael Höhnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht Köln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln executive board: Kadir Akin Dr. Michael Höhnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497 **************************************************************************** ------=_NextPartTM-000-269ed284-0ba0-45cc-bfe3-0742919cf61b--

On Jan 9, 2014, at 3:16 AM, Markus Stockhausen <stockhausen@collogia.de> wrote:
We see a quite a heavy latency penalty using KVM VirtIO disks in comparison to ESX. Doing one I/O onto disk inside a VM usually adds 370us of overhead in the virtualisation layer. This has been tested with VirtIO-SCSI and windows guest (2K3). More here (still now answer yet):
It would be interesting to see you do the same tests on virtio-blk instead of virtio-scsi.
participants (3)
-
Blaster
-
Markus Stockhausen
-
Sander Grendelman