10:30 a.m.
This is a multi-part message in MIME format.
--------------000301030102040906030909
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 8bit
On 02/13/2015 08:20 PM, Markus Stockhausen wrote:
> Von: users-bounces(a)ovirt.org [users-bounces(a)ovirt.org]"
im Auftrag von "Darrell Budic [budic(a)onholyground.com]
> Gesendet: Freitag, 13. Februar 2015 19:03
> An: Nicolas Ecarnot
> Cc: users
> Betreff: Re: [ovirt-users] How long do your migrations last?
>
> Im under the impression it depends more on the hosts memory assignment
> than disk size. libvirt has to synchronize that over your networking setup. Your
> times sound like mine over 1G ethernet with a 9000 MTU, most of my machines
> are 1-4GB ram. Ive another setup with a 10G backend that can migrate larger
> machines much faster. Things that do a lot of memory access (databases, say)
> or use more of their allocated memory, tend to take longer to migrate as its
> more work for libvirt to get it synchronized.
>
> A 10G+ backend is the best way to speed this up, and there are libvirt variables
> you can tweak to allocate more bandwidth to a migration (and the # of simultaneous
> migrations you allow). I think the defaults are 3 at max of 30% of your available
> bandwidth. I dont think this takes bonds into account, so if you have bonded
> connections, you may be able to allocate more % or allow more simultaneous
> migrations. Keep in mind that if youre sharing bandwidth/media with iSCSI, that
> some bandwidth will be needed there as well, how much depends on your storage
> load. A dedicated NIC could definitely help, especially if youre trying to tune
libvirt for this.
>
> -Darrell
>
>> On Feb 13, 2015, at 8:53 AM, Nicolas Ecarnot <nicolas(a)ecarnot.net> wrote:
>>
>> Hello list,
>>
>> Our storage domains are iSCSI on dedicated network, and when migrating VMs, the
duration varies according to the size of the vDisks.
>>
>> The smallest VMs are migrated in about 20 seconds, while the biggest one may take
more than 5 or 10 minutes.
>> The average duration is 90 seconds.
>>
>> Questions :
>>
>> 1- Though I may have understood that the task of migration was made by the SPM, I
don't know what it actually does? (which bytes goes where)
>>
>> 2- Do our times sound OK, or does it look like improvable?
>>
>> 3- What bottleneck should I investigate? I'm thinking about the dedicated
hardware NICs setup of the hosts, the SAN, the MTU has already been setup at 9000...
>>
>> Any ideas welcomed.
>>
>> --
> Nicolas Ecarnot
If we speak about migration of VMs - relocating qemu process -
than speed depends mostly on memory change pressure. The
more changes per second the more restart the process needs.
Best solution to speed it up is to enlarge migration_max_bandwidth
in /etc/vdsm/vdsm.conf from default 30MB/s to something higher.
We use 150Mb/s in 10Gbit network. With default we have seen
migrations that will not come to an end.
+1
moreover, upstream qemu have some more ways to speed this up
- post migration copy (a.k.a "user page faults") - basically migrate
immediate to the dest and copy mem pages from source
- migration over rdma
- migration throttling -
http://lists.gnu.org/archive/html/qemu-devel/2013-05/msg00040.html
When talking about disks. It depends on how many disks you
have attached to a single VM. The more disks and the more
similar their sizes they are the faster you can migrate/operate
on them.
For example take a SAP system with 3 disks of 20GB system
20 GB executables and 300GB database. When issung disk
operations (like snapshots) they will start in parallel for each disk.
Disk operations will finish earlier for smaller disks. So in the end
you will have only one operation left that may take hours.
E.g. delete snapshot will start at ~220MB/s when running with
three disks and end at ~60MB/s when only one disk snapshot
deletion is active.
Best regards.
Markus=
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--------------000301030102040906030909
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 02/13/2015 08:20 PM, Markus
Stockhausen wrote:<br>
</div>
<blockquote
cite="mid:12EF8D94C6F8734FB2FF37B9FBEDD1735F9DE418@EXCHANGE.collogia.de"
type="cite">
<blockquote type="cite">
<pre wrap="">Von: <a class="moz-txt-link-abbreviated"
href="mailto:users-bounces@ovirt.org">users-bounces@ovirt.org</a>
[<a class="moz-txt-link-abbreviated"
href="mailto:users-bounces@ovirt.org">users-bounces@ovirt.org</a>]&quot;
im Auftrag von &quot;Darrell Budic [<a
class="moz-txt-link-abbreviated"
href="mailto:budic@onholyground.com">budic@onholyground.com</a>]
Gesendet: Freitag, 13. Februar 2015 19:03
An: Nicolas Ecarnot
Cc: users
Betreff: Re: [ovirt-users] How long do your migrations last?
Im under the impression it depends more on the hosts memory assignment
than disk size. libvirt has to synchronize that over your networking setup. Your
times sound like mine over 1G ethernet with a 9000 MTU, most of my machines
are 1-4GB ram. Ive another setup with a 10G backend that can migrate larger
machines much faster. Things that do a lot of memory access (databases, say)
or use more of their allocated memory, tend to take longer to migrate as its
more work for libvirt to get it synchronized.
A 10G+ backend is the best way to speed this up, and there are libvirt variables
you can tweak to allocate more bandwidth to a migration (and the # of simultaneous
migrations you allow). I think the defaults are 3 at max of 30% of your available
bandwidth. I dont think this takes bonds into account, so if you have bonded
connections, you may be able to allocate more % or allow more simultaneous
migrations. Keep in mind that if youre sharing bandwidth/media with iSCSI, that
some bandwidth will be needed there as well, how much depends on your storage
load. A dedicated NIC could definitely help, especially if youre trying to tune libvirt
for this.
-Darrell
</pre>
<blockquote type="cite">
<pre wrap="">On Feb 13, 2015, at 8:53 AM, Nicolas Ecarnot <a
class="moz-txt-link-rfc2396E"
href="mailto:nicolas@ecarnot.net"><nicolas@ecarnot.net></a>
wrote:
Hello list,
Our storage domains are iSCSI on dedicated network, and when migrating VMs, the duration
varies according to the size of the vDisks.
The smallest VMs are migrated in about 20 seconds, while the biggest one may take more
than 5 or 10 minutes.
The average duration is 90 seconds.
Questions :
1- Though I may have understood that the task of migration was made by the SPM, I
don't know what it actually does? (which bytes goes where)
2- Do our times sound OK, or does it look like improvable?
3- What bottleneck should I investigate? I'm thinking about the dedicated hardware
NICs setup of the hosts, the SAN, the MTU has already been setup at 9000...
Any ideas welcomed.
--
</pre>
</blockquote>
<pre wrap="">Nicolas Ecarnot
</pre>
</blockquote>
<pre wrap="">
If we speak about migration of VMs - relocating qemu process -
than speed depends mostly on memory change pressure. The
more changes per second the more restart the process needs.
Best solution to speed it up is to enlarge migration_max_bandwidth
in /etc/vdsm/vdsm.conf from default 30MB/s to something higher.
We use 150Mb/s in 10Gbit network. With default we have seen
migrations that will not come to an end.
</pre>
</blockquote>
+1<br>
<br>
moreover, upstream qemu have some more ways to speed this up<br>
<br>
- post migration copy (a.k.a "user page faults") - basically
migrate immediate to the dest and copy mem pages from source<br>
- migration over rdma <br>
- migration throttling - <a
href="http://lists.gnu.org/archive/html/qemu-devel/2013-05/msg00040....
<br>
<br>
<blockquote
cite="mid:12EF8D94C6F8734FB2FF37B9FBEDD1735F9DE418@EXCHANGE.collogia.de"
type="cite">
<pre wrap="">
When talking about disks. It depends on how many disks you
have attached to a single VM. The more disks and the more
similar their sizes they are the faster you can migrate/operate
on them.
For example take a SAP system with 3 disks of 20GB system
20 GB executables and 300GB database. When issung disk
operations (like snapshots) they will start in parallel for each disk.
Disk operations will finish earlier for smaller disks. So in the end
you will have only one operation left that may take hours.
E.g. delete snapshot will start at ~220MB/s when running with
three disks and end at ~60MB/s when only one disk snapshot
deletion is active.
Best regards.
Markus=</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
<br>
</body>
</html>
--------------000301030102040906030909--