[ovirt-users] massive simultaneous vms migrations ?

Wed Apr 19 09:21:53 UTC 2017

Hi yaniv, 

Thanks for your response. My answers below. 

cordialement, regards, 

	Nelson LAMEIRAS 
Ingénieur Systèmes et Réseaux / Systems and Networks engineer 
Tel: +33 5 32 09 09 70 
nelson.lameiras at lyra-network.com 

www.lyra-network.com | www.payzen.eu 

Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE 

From: "Yaniv Kaul" <ykaul at redhat.com> 
To: "Nelson Lameiras" <nelson.lameiras at lyra-network.com> 
Cc: "ovirt users" <users at ovirt.org> 
Sent: Wednesday, April 19, 2017 9:13:27 AM 
Subject: Re: [ovirt-users] massive simultaneous vms migrations ? 

On Tue, Apr 18, 2017 at 7:48 PM, Nelson Lameiras < nelson.lameiras at lyra-network.com > wrote: 

hello, 

When putting a host on "maintenance mode", all vms start migrating to other hosts. 

We have some hosts that have 60 vms. So this will create a 60 vms migrating simultaneously. 
Some vms are under so much heavy loads that migration fails often (our guess is that massive simultaneous migrations does not help migration convergence) - even with "suspend workload if needed" migraton policy. 

- Does oVirt really launches 60 simultaneous migrations or is there a queuing system ? 
- If there is a queuing system, is there a way to configure a maximum number of simultaneous migrations ? 

I did see a "migration bandwidth limit", but this is quite what we are looking for. 

What migration policy are you using? 
-- "suspend workkiad if needed" 
Are you using a dedicated migration network, or the ovirtmgmt network? 
-- ovirtmgmt network 

BQ_BEGIN

my setup: 
ovirt-engine +hosted engine 4.1.1 
hosts : centos 7.3 fully updated. 

for full context to understand this question : 2 times in the past, when trying to put a host in maintenance, host stopped responding during massive migrations and was fenced by engine. It's still unclear why host stopped responding, but we think that migrating 60+ vms simultaneously puts a heavy strain on storage ? So we would like to better control migration process in order to better understand what's happening. This scenario is "production only" since our labs do not contain nearly as much vm with such heavy loads. So rather than trying to reproduce, we are trying to avoid ;) 

BQ_END

If you could open a bug with relevant logs on the host not responding, that'd be great. 
-- Too late, we made the "precipitaded" mistake of reinstalling server (beause of other reasons), so all traces are lost. Next time I will make sure to keep traces. 
Live migration doesn't touch the storage (disks re not moving anywhere), but it does stress the network. I doubt it, but perhaps you over-saturate the ovirtmgmt network. 
-- This makes sens. It's a long shot though. Maybe it would be possible to create a VLAN dedicated to migrations in same physical network et use QOS to reserve always some bandwith to oVirt administration ? (maybe opening can of worms here....) 
Y. 

BQ_BEGIN

cordialement, regards, 

	Nelson LAMEIRAS 
Ingénieur Systèmes et Réseaux / Systems and Networks engineer 
Tel: +33 5 32 09 09 70 

nelson.lameiras at lyra-network.com 

www.lyra-network.com | www.payzen.eu 

Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE 

_______________________________________________ 
Users mailing list 
Users at ovirt.org 
http://lists.ovirt.org/mailman/listinfo/users 

BQ_END

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170419/7797a056/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: element-signature_logo_lyra_115x94.jpg
Type: image/jpeg
Size: 3846 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170419/7797a056/attachment-0010.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: element-signature_logo_YouTube_32x28.jpg
Type: image/jpeg
Size: 1604 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170419/7797a056/attachment-0011.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: element-signature_logo_LinkedIn_41x28.jpg
Type: image/jpeg
Size: 1635 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170419/7797a056/attachment-0012.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: element-signature_logo_Twitter_42x28.jpg
Type: image/jpeg
Size: 1629 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170419/7797a056/attachment-0013.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: element-signature_payzen_61x28.jpg
Type: image/jpeg
Size: 1864 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170419/7797a056/attachment-0014.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: element-signature_logo_lyra_115x94.jpg
Type: image/jpeg
Size: 3846 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170419/7797a056/attachment-0015.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: element-signature_logo_YouTube_32x28.jpg
Type: image/jpeg
Size: 1604 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170419/7797a056/attachment-0016.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: element-signature_logo_LinkedIn_41x28.jpg
Type: image/jpeg
Size: 1635 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170419/7797a056/attachment-0017.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: element-signature_logo_Twitter_42x28.jpg
Type: image/jpeg
Size: 1629 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170419/7797a056/attachment-0018.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: element-signature_payzen_61x28.jpg
Type: image/jpeg
Size: 1864 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170419/7797a056/attachment-0019.jpg>