4:31 p.m.
Ben <gravyfish(a)gmail.com> writes:
Hi Milan,
Thanks for your reply. I checked the firewall, and saw that both the bond0
interface and the VLAN interface bond0.20 had been added to the default
zone, which I believe should provide the necessary firewall access (output
below)
I double-checked the destination host's VDSM logs and wasn't able to find
any warning or error-level logs during the migration timeframe.
I checked the migration_port_* and *_port settings in qemu.conf and
libvirtd.conf and all lines are commented. I have not modified either file.
The commented out settings define the default port used for migrations,
so they are valid even when commented out. I can see you have
libvirt-tls open below, not sure about the QEMU ports. If migration
works when not using a separate migration network then it should work
with the same rules for the migration network, so I think your settings
are OK.
The fact that you don't get any better explanation than "unexpectedly
failed" and that it fails before transferring any data indicates a
possible networking error, but I can't help with that, someone with
networking knowledge should.
You can also try to enable libvirt debugging on both the sides in
/etc/libvirt/libvirtd.conf and restart libvirt (beware, those logs are
huge). libvirt logs should report some error.
[root@vhost2 vdsm]# firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: bond0 bond0.20 em1 em2 migration ovirtmgmt p1p1
sources:
services: cockpit dhcpv6-client libvirt-tls ovirt-imageio ovirt-vmconsole
snmp ssh vdsm
ports: 1311/tcp 22/tcp 6081/udp 5666/tcp
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
On Mon, Jan 20, 2020 at 6:29 AM Milan Zamazal <mzamazal(a)redhat.com> wrote:
> Ben <gravyfish(a)gmail.com> writes:
>
> > Hi, I'm pretty stuck at the moment so I hope someone can help me.
> >
> > I have an oVirt 4.3 data center with two hosts. Recently, I attempted to
> > segregate migration traffic from the the standard ovirtmgmt network,
> where
> > the VM traffic and all other traffic resides.
> >
> > I set up the VLAN on my router and switch, and created LACP bonds on both
> > hosts, tagging them with the VLAN ID. I confirmed the routes work fine,
> and
> > traffic speeds are as expected. MTU is set to 9000.
> >
> > After configuring the migration network in the cluster and dragging and
> > dropping it onto the bonds on each host, VMs fail to migrate.
> >
> > oVirt is not reporting any issues with the network interfaces or sync
> with
> > the hosts. However, when I attempt to live-migrate a VM, progress gets to
> > 1% and stalls. The transfer rate is 0Mbps, and the operation eventually
> > fails.
> >
> > I have not been able to identify anything useful in the VDSM logs on the
> > source or destination hosts, or in the engine logs. It repeats the below
> > WARNING and INFO logs for the duration of the process, then logs the last
> > entries when it fails. I can provide more logs if it would help. I'm not
> > even sure where to start -- since I am a novice at networking, at best,
> my
> > suspicion the entire time was that something is misconfigured in my
> > network. However, the routes are good, speed tests are fine, and I can't
> > find anything else wrong with the connections. It's not impacting any
> other
> > traffic over the bond interfaces.
> >
> > Are there other requirements that must be met for VMs to migrate over a
> > separate interface/network?
>
> Hi, did you check your firewall settings? Are the required ports open?
> See migration_port_* options in /etc/libvirt/qemu.conf and *_port
> options in /etc/libvirt/libvirtd.conf.
>
> Is there any error reported in the destination vdsm.log?
>
> Regards,
> Milan
>
> > 2020-01-12 03:18:28,245-0500 WARN (migmon/a24fd7e3) [virt.vm]
> > (vmId='a24fd7e3-161c-451e-8880-b3e7e1f7d86f') Migration stalling:
> remaining
> > (4191MiB) > lowmark (4191MiB). (migration:854)
> > 2020-01-12 03:18:28,245-0500 INFO (migmon/a24fd7e3) [virt.vm]
> > (vmId='a24fd7e3-161c-451e-8880-b3e7e1f7d86f') Migration Progress:
930.341
> > seconds elapsed, 1% of data processed, total data: 4192MB, processed
> data:
> > 0MB, remaining data: 4191MB, transfer speed 0MBps, zero pages: 149MB,
> > compressed: 0MB, dirty rate: 0, memory iteration: 1 (migration:881)
> > 2020-01-12 03:18:31,386-0500 ERROR (migsrc/a24fd7e3) [virt.vm]
> > (vmId='a24fd7e3-161c-451e-8880-b3e7e1f7d86f') operation failed:
migration
> > out job: unexpectedly failed (migration:282)
> > 2020-01-12 03:18:32,695-0500 ERROR (migsrc/a24fd7e3) [virt.vm]
> > (vmId='a24fd7e3-161c-451e-8880-b3e7e1f7d86f') Failed to migrate
> > (migration:450)
> > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py",
line
> 431,
> > in _regular_run
> > time.time(), migrationParams, machineParams
> > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py",
line
> 505,
> > in _startUnderlyingMigration
> > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py",
line
> 591,
> > in _perform_with_conv_schedule
> > self._perform_migration(duri, muri)
> > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py",
line
> 525,
> > in _perform_migration
> > self._migration_flags)
> > libvirtError: operation failed: migration out job: unexpectedly failed
> > 2020-01-12 03:18:40,880-0500 INFO (jsonrpc/6) [api.virt] FINISH
> > getMigrationStatus return={'status': {'message': 'Done',
'code': 0},
> > 'migrationStats': {'status': {'message': 'Fatal
error during migration',
> > 'code': 12}, 'progress': 1L}} from=::ffff:10.0.0.20,41462,
> > vmId=a24fd7e3-161c-451e-8880-b3e7e1f7d86f (api:54)
> > _______________________________________________
> > Users mailing list -- users(a)ovirt.org
> > To unsubscribe send an email to users-leave(a)ovirt.org
> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> > oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> > List Archives:
> >
>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/PB3TQTFXWKA...
>
>