Migration Failure Due to network

Here are the two Events I can see : Host Hypervisor has network interface which exceeded the defined threshold [95%] (em1: transmit rate[98%], receive rate [4%]) Migration failed (VM: Clone-ubuntu, Source: Hypervisor, Destination: hypervisor22). Any ideas ? It it not the first time I have this kind of network saturation issue... My current temporary setup is this : 2 hosts with only one network physical interface used and assigned to ovirtmgmt, used by all VM. Do migration of a running VM from one host to the other use the network assigned to that VM for migration ? Or does it use ovirtmgmt in all cases ? What minimum network architecture setup would you advice me to avoid this kind of issues ?

Alexis, Use "Clusters" -> "Logical Networks" -> "Manage Networks" for assigning network for migrations. It's ovirtmgmt by default. Note that migration network has to have IPs on hosts. Hosts have VDSM configuration option - [vars]/migration_max_bandwidth in /etc/vdsm/vdsm.conf. It's 52MB/s by default. So it looks like your network is already highly used. Your VM migration fails because migration is so slow so it just can't migrate without downtime. You can adjust "Use custom migration downtime" VM option to make any VM migrated over any network. Please note it's a maximal value. Actually it starts with a value 10 times lower and increases it automatically. 5000 is good value to start in your case. On Tue, 2016-07-19 at 17:44 +0200, Alexis HAUSER wrote:
Here are the two Events I can see : Host Hypervisor has network interface which exceeded the defined threshold [95%] (em1: transmit rate[98%], receive rate [4%]) Migration failed (VM: Clone-ubuntu, Source: Hypervisor, Destination: hypervisor22).
Any ideas ?
It it not the first time I have this kind of network saturation issue... My current temporary setup is this : 2 hosts with only one network physical interface used and assigned to ovirtmgmt, used by all VM. Do migration of a running VM from one host to the other use the network assigned to that VM for migration ? Or does it use ovirtmgmt in all cases ?
What minimum network architecture setup would you advice me to avoid this kind of issues ?
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

Use "Clusters" -> "Logical Networks" -> "Manage Networks" for assigning network for migrations. It's ovirtmgmt by default. Note that migration network has to have IPs on hosts.
Nice, do you think I should dedicate a link only for migration, for safety ?
Hosts have VDSM configuration option - [vars]/migration_max_bandwidth in /etc/vdsm/vdsm.conf. It's 52MB/s by default. So it looks like your network is already highly used.
It's not highly used, it's yet on a 10/100 Mb switch...But it's just temporary. Anyway it's interesting because it simulates a highly use network, which can happen.
Your VM migration fails because migration is so slow so it just can't migrate without downtime. You can adjust "Use custom migration downtime" VM option to make any VM migrated over any network. Please note it's a maximal value. Actually it starts with a value 10 times lower and increases it automatically. 5000 is good value to start in your case.
Thanks, but it doesn't seem to change anything : I added max_outgoing_migrations = 1 migration_max_bandwidth = 5 in /etc/vdsm/vdsm.conf on my concerned host, but it still goes to 98% use of network and migration fail...I restart vdsmd.service but it doesn't seem to change anything. Any idea ?

On 20 Jul 2016, at 15:22, Alexis HAUSER <alexis.hauser@telecom-bretagne.eu> wrote:
Use "Clusters" -> "Logical Networks" -> "Manage Networks" for assigning network for migrations. It's ovirtmgmt by default. Note that migration network has to have IPs on hosts.
Nice, do you think I should dedicate a link only for migration, for safety ?
Hosts have VDSM configuration option - [vars]/migration_max_bandwidth in /etc/vdsm/vdsm.conf. It's 52MB/s by default. So it looks like your network is already highly used.
It's not highly used, it's yet on a 10/100 Mb switch...But it's just temporary. Anyway it's interesting because it simulates a highly use network, which can happen.
Your VM migration fails because migration is so slow so it just can't migrate without downtime. You can adjust "Use custom migration downtime" VM option to make any VM migrated over any network. Please note it's a maximal value. Actually it starts with a value 10 times lower and increases it automatically. 5000 is good value to start in your case.
Thanks, but it doesn't seem to change anything : I added max_outgoing_migrations = 1 migration_max_bandwidth = 5
well this is not going to make it any faster, quite the opposite:) You need to play with the downtime setting (in UI or in vdsm.conf) to affect convergence in your constrained connectivity setup There’s a lot more intelligence in oVirt 4.0 regarding this, so if it’s an option I would recommend to upgrade and change the migration policy to one of the new ones Thanks, michal
in /etc/vdsm/vdsm.conf on my concerned host, but it still goes to 98% use of network and migration fail...I restart vdsmd.service but it doesn't seem to change anything. Any idea ? _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (3)
-
Alexis HAUSER
-
Michal Skrivanek
-
Pavel Gashev