6:26 a.m.
Hi, I'm pretty stuck at the moment so I hope someone can help me.
I have an oVirt 4.3 data center with two hosts. Recently, I attempted to
segregate migration traffic from the the standard ovirtmgmt network, where
the VM traffic and all other traffic resides.
I set up the VLAN on my router and switch, and created LACP bonds on both
hosts, tagging them with the VLAN ID. I confirmed the routes work fine, and
traffic speeds are as expected. MTU is set to 9000.
After configuring the migration network in the cluster and dragging and
dropping it onto the bonds on each host, VMs fail to migrate.
oVirt is not reporting any issues with the network interfaces or sync with
the hosts. However, when I attempt to live-migrate a VM, progress gets to
1% and stalls. The transfer rate is 0Mbps, and the operation eventually
fails.
I have not been able to identify anything useful in the VDSM logs on the
source or destination hosts, or in the engine logs. It repeats the below
WARNING and INFO logs for the duration of the process, then logs the last
entries when it fails. I can provide more logs if it would help. I'm not
even sure where to start -- since I am a novice at networking, at best, my
suspicion the entire time was that something is misconfigured in my
network. However, the routes are good, speed tests are fine, and I can't
find anything else wrong with the connections. It's not impacting any other
traffic over the bond interfaces.
Are there other requirements that must be met for VMs to migrate over a
separate interface/network?
2020-01-12 03:18:28,245-0500 WARN (migmon/a24fd7e3) [virt.vm]
(vmId='a24fd7e3-161c-451e-8880-b3e7e1f7d86f') Migration stalling: remaining
(4191MiB) > lowmark (4191MiB). (migration:854)
2020-01-12 03:18:28,245-0500 INFO (migmon/a24fd7e3) [virt.vm]
(vmId='a24fd7e3-161c-451e-8880-b3e7e1f7d86f') Migration Progress: 930.341
seconds elapsed, 1% of data processed, total data: 4192MB, processed data:
0MB, remaining data: 4191MB, transfer speed 0MBps, zero pages: 149MB,
compressed: 0MB, dirty rate: 0, memory iteration: 1 (migration:881)
2020-01-12 03:18:31,386-0500 ERROR (migsrc/a24fd7e3) [virt.vm]
(vmId='a24fd7e3-161c-451e-8880-b3e7e1f7d86f') operation failed: migration
out job: unexpectedly failed (migration:282)
2020-01-12 03:18:32,695-0500 ERROR (migsrc/a24fd7e3) [virt.vm]
(vmId='a24fd7e3-161c-451e-8880-b3e7e1f7d86f') Failed to migrate
(migration:450)
File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 431,
in _regular_run
time.time(), migrationParams, machineParams
File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 505,
in _startUnderlyingMigration
File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 591,
in _perform_with_conv_schedule
self._perform_migration(duri, muri)
File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 525,
in _perform_migration
self._migration_flags)
libvirtError: operation failed: migration out job: unexpectedly failed
2020-01-12 03:18:40,880-0500 INFO (jsonrpc/6) [api.virt] FINISH
getMigrationStatus return={'status': {'message': 'Done',
'code': 0},
'migrationStats': {'status': {'message': 'Fatal error during
migration',
'code': 12}, 'progress': 1L}} from=::ffff:10.0.0.20,41462,
vmId=a24fd7e3-161c-451e-8880-b3e7e1f7d86f (api:54)