Hi, Just trying to place in maintenance mode for a version upgrade, and one VM fails to migrate. The other 20-odd move over successfully. In /var/log/libvirt/qemu/, the VM's log on the source reports: 2017-05-16 09:48:06.339+0000: initiating migration 2017-05-16T09:52:25.498932Z qemu-kvm: socket_writev_buffer: Got err=32 for (131328/18446744073709551615) 2017-05-16 09:52:47.311+0000: initiating migration 2017-05-16T09:57:06.755402Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 09:57:55.109+0000: initiating migration 2017-05-16T10:02:14.143221Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:02:59.497+0000: initiating migration 2017-05-16T10:07:18.542872Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:08:03.896+0000: initiating migration 2017-05-16T10:12:23.206731Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:13:08.941+0000: initiating migration 2017-05-16T10:17:27.861843Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:18:13.690+0000: initiating migration 2017-05-16T10:22:32.929689Z qemu-kvm: socket_writev_buffer: Got err=32 for (69803/18446744073709551615) 2017-05-16 10:23:19.846+0000: initiating migration 2017-05-16T10:27:39.175724Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:28:25.141+0000: initiating migration 2017-05-16T10:28:35.620070Z qemu-kvm: socket_writev_buffer: Got err=32 for (65753/18446744073709551615) 2017-05-16 10:29:10.678+0000: initiating migration 2017-05-16T10:33:29.718527Z qemu-kvm: socket_writev_buffer: Got err=32 for (53477/18446744073709551615) 2017-05-16 10:38:35.517+0000: initiating migration

Hi Francesco, I left it running after I posted to the list, and it eventually (after many failed attempts) moved the VM without any intervention by me, and then updated the host, so that explains the differences in the versions of qemu between the hosts (they probably would have been the same when I tried the move first). The xml is attached. qemu and libvirt versions on the source host: ipxe-roms-qemu-20160127-5.git6366fa7a.el7.noarch libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64 qemu-img-ev-2.6.0-28.el7_3.9.1.x86_64 qemu-kvm-common-ev-2.6.0-28.el7_3.9.1.x86_64 qemu-kvm-ev-2.6.0-28.el7_3.9.1.x86_64 qemu-kvm-tools-ev-2.6.0-28.el7_3.3.1.x86_64 libvirt-2.0.0-10.el7_3.5.x86_64 libvirt-client-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-config-network-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-config-nwfilter-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-interface-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-lxc-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-network-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-nodedev-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-nwfilter-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-secret-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-storage-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-kvm-2.0.0-10.el7_3.5.x86_64 libvirt-lock-sanlock-2.0.0-10.el7_3.5.x86_64 libvirt-python-2.0.0-2.el7.x86_64 qemu and libvirt versions on the dest host: ipxe-roms-qemu-20160127-5.git6366fa7a.el7.noarch libvirt-daemon-driver-qemu-2.0.0-10.el7_3.4.x86_64 qemu-img-ev-2.6.0-28.el7_3.3.1.x86_64 qemu-kvm-common-ev-2.6.0-28.el7_3.3.1.x86_64 qemu-kvm-ev-2.6.0-28.el7_3.3.1.x86_64 qemu-kvm-tools-ev-2.6.0-28.el7_3.3.1.x86_64 libvirt-client-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-config-nwfilter-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-interface-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-network-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-nodedev-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-nwfilter-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-qemu-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-secret-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-storage-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-kvm-2.0.0-10.el7_3.4.x86_64 libvirt-lock-sanlock-2.0.0-10.el7_3.4.x86_64 libvirt-python-2.0.0-2.el7.x86_64 Thanks, Cam On Wed, May 17, 2017 at 9:12 AM, Francesco Romani <fromani(a)redhat.com&gt; wrote: > > On 05/16/2017 01:06 PM, cmc wrote: >> Hi, >> >> Just trying to place in maintenance mode for a version upgrade, and >> one VM fails to migrate. The other 20-odd move over successfully. In >> /var/log/libvirt/qemu/, the VM's log on the source reports: >> >> 2017-05-16 09:48:06.339+0000: initiating migration >> 2017-05-16T09:52:25.498932Z qemu-kvm: socket_writev_buffer: Got err=32 >> for (131328/18446744073709551615) >> 2017-05-16 09:52:47.311+0000: initiating migration >> 2017-05-16T09:57:06.755402Z qemu-kvm: socket_writev_buffer: Got err=32 >> for (69776/18446744073709551615) >> 2017-05-16 09:57:55.109+0000: initiating migration >> 2017-05-16T10:02:14.143221Z qemu-kvm: socket_writev_buffer: Got err=32 >> for (69776/18446744073709551615) >> 2017-05-16 10:02:59.497+0000: initiating migration >> 2017-05-16T10:07:18.542872Z qemu-kvm: socket_writev_buffer: Got err=32 >> for (69776/18446744073709551615) >> 2017-05-16 10:08:03.896+0000: initiating migration >> 2017-05-16T10:12:23.206731Z qemu-kvm: socket_writev_buffer: Got err=32 >> for (69776/18446744073709551615) >> 2017-05-16 10:13:08.941+0000: initiating migration >> 2017-05-16T10:17:27.861843Z qemu-kvm: socket_writev_buffer: Got err=32 >> for (69776/18446744073709551615) >> 2017-05-16 10:18:13.690+0000: initiating migration >> 2017-05-16T10:22:32.929689Z qemu-kvm: socket_writev_buffer: Got err=32 >> for (69803/18446744073709551615) >> 2017-05-16 10:23:19.846+0000: initiating migration >> 2017-05-16T10:27:39.175724Z qemu-kvm: socket_writev_buffer: Got err=32 >> for (69776/18446744073709551615) >> 2017-05-16 10:28:25.141+0000: initiating migration >> 2017-05-16T10:28:35.620070Z qemu-kvm: socket_writev_buffer: Got err=32 >> for (65753/18446744073709551615) >> 2017-05-16 10:29:10.678+0000: initiating migration >> 2017-05-16T10:33:29.718527Z qemu-kvm: socket_writev_buffer: Got err=32 >> for (53477/18446744073709551615) >> 2017-05-16 10:38:35.517+0000: initiating migration >> > > Hi, > it seems either qemu issue or misconfiguration. To investigate, we need > more data; so could you please share: > 1. the domain XML (virtsh -r dumpxml ...) and/or the qemu command line > of the affected VM, on the source side > 2. the version of QEMU and libvirt that you are running > > Thanks and bests, > > -- > Francesco Romani > Senior SW Eng., Virtualization R&D > Red Hat > IRC: fromani github: @fromanirh > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users

I have to shut the VM down to stop it trying to repeatedly trying to migrate the problematic host (always the same one). If I take it out of maintenance, it will move the VMs back to balance (as per policy), so this is rather inconvenient. It took leaving it overnight and letting oVirt try repeatedly every few minutes to get it to migrate the VM (I can't wait that long, so I've had to shut that VM down for now)

From https://bugzilla.redhat.com/show_bug.cgi?id=1355662 , however, we

From oVirt perspective, it seems all good. We need to investigate the

On Wed, May 17, 2017 at 4:13 PM, cmc <iucounu(a)gmail.com&gt; wrote: > Just a note on this: a similar thing is now happening with the same VM > when I upgrade the other node, i.e., it can't move this one VM over > (so far) from one host to another. I will leave it trying overnight to > see if it succeeds. > > Thanks, > > Cam > > On Wed, May 17, 2017 at 11:40 AM, cmc <iucounu(a)gmail.com&gt; wrote: >> Hi Francesco, >> >> I left it running after I posted to the list, and it eventually (after >> many failed attempts) moved the VM without any intervention by me, and >> then updated the host, so that explains the differences in the >> versions of qemu between the hosts (they probably would have been the >> same when I tried the move first). The xml is attached. >> >> qemu and libvirt versions on the source host: >> >> ipxe-roms-qemu-20160127-5.git6366fa7a.el7.noarch >> libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64 >> qemu-img-ev-2.6.0-28.el7_3.9.1.x86_64 >> qemu-kvm-common-ev-2.6.0-28.el7_3.9.1.x86_64 >> qemu-kvm-ev-2.6.0-28.el7_3.9.1.x86_64 >> qemu-kvm-tools-ev-2.6.0-28.el7_3.3.1.x86_64 >> >> libvirt-2.0.0-10.el7_3.5.x86_64 >> libvirt-client-2.0.0-10.el7_3.5.x86_64 >> libvirt-daemon-2.0.0-10.el7_3.5.x86_64 >> libvirt-daemon-config-network-2.0.0-10.el7_3.5.x86_64 >> libvirt-daemon-config-nwfilter-2.0.0-10.el7_3.5.x86_64 >> libvirt-daemon-driver-interface-2.0.0-10.el7_3.5.x86_64 >> libvirt-daemon-driver-lxc-2.0.0-10.el7_3.5.x86_64 >> libvirt-daemon-driver-network-2.0.0-10.el7_3.5.x86_64 >> libvirt-daemon-driver-nodedev-2.0.0-10.el7_3.5.x86_64 >> libvirt-daemon-driver-nwfilter-2.0.0-10.el7_3.5.x86_64 >> libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64 >> libvirt-daemon-driver-secret-2.0.0-10.el7_3.5.x86_64 >> libvirt-daemon-driver-storage-2.0.0-10.el7_3.5.x86_64 >> libvirt-daemon-kvm-2.0.0-10.el7_3.5.x86_64 >> libvirt-lock-sanlock-2.0.0-10.el7_3.5.x86_64 >> libvirt-python-2.0.0-2.el7.x86_64 >> >> qemu and libvirt versions on the dest host: >> >> ipxe-roms-qemu-20160127-5.git6366fa7a.el7.noarch >> libvirt-daemon-driver-qemu-2.0.0-10.el7_3.4.x86_64 >> qemu-img-ev-2.6.0-28.el7_3.3.1.x86_64 >> qemu-kvm-common-ev-2.6.0-28.el7_3.3.1.x86_64 >> qemu-kvm-ev-2.6.0-28.el7_3.3.1.x86_64 >> qemu-kvm-tools-ev-2.6.0-28.el7_3.3.1.x86_64 >> >> libvirt-client-2.0.0-10.el7_3.4.x86_64 >> libvirt-daemon-2.0.0-10.el7_3.4.x86_64 >> libvirt-daemon-config-nwfilter-2.0.0-10.el7_3.4.x86_64 >> libvirt-daemon-driver-interface-2.0.0-10.el7_3.4.x86_64 >> libvirt-daemon-driver-network-2.0.0-10.el7_3.4.x86_64 >> libvirt-daemon-driver-nodedev-2.0.0-10.el7_3.4.x86_64 >> libvirt-daemon-driver-nwfilter-2.0.0-10.el7_3.4.x86_64 >> libvirt-daemon-driver-qemu-2.0.0-10.el7_3.4.x86_64 >> libvirt-daemon-driver-secret-2.0.0-10.el7_3.4.x86_64 >> libvirt-daemon-driver-storage-2.0.0-10.el7_3.4.x86_64 >> libvirt-daemon-kvm-2.0.0-10.el7_3.4.x86_64 >> libvirt-lock-sanlock-2.0.0-10.el7_3.4.x86_64 >> libvirt-python-2.0.0-2.el7.x86_64 >> >> >> Thanks, >> >> Cam >> >> On Wed, May 17, 2017 at 9:12 AM, Francesco Romani <fromani(a)redhat.com&gt; wrote: >>> On 05/16/2017 01:06 PM, cmc wrote: >>>> Hi, >>>> >>>> Just trying to place in maintenance mode for a version upgrade, and >>>> one VM fails to migrate. The other 20-odd move over successfully. In >>>> /var/log/libvirt/qemu/, the VM's log on the source reports: >>>> >>>> 2017-05-16 09:48:06.339+0000: initiating migration >>>> 2017-05-16T09:52:25.498932Z qemu-kvm: socket_writev_buffer: Got err=32 >>>> for (131328/18446744073709551615) >>>> 2017-05-16 09:52:47.311+0000: initiating migration >>>> 2017-05-16T09:57:06.755402Z qemu-kvm: socket_writev_buffer: Got err=32 >>>> for (69776/18446744073709551615) >>>> 2017-05-16 09:57:55.109+0000: initiating migration >>>> 2017-05-16T10:02:14.143221Z qemu-kvm: socket_writev_buffer: Got err=32 >>>> for (69776/18446744073709551615) >>>> 2017-05-16 10:02:59.497+0000: initiating migration >>>> 2017-05-16T10:07:18.542872Z qemu-kvm: socket_writev_buffer: Got err=32 >>>> for (69776/18446744073709551615) >>>> 2017-05-16 10:08:03.896+0000: initiating migration >>>> 2017-05-16T10:12:23.206731Z qemu-kvm: socket_writev_buffer: Got err=32 >>>> for (69776/18446744073709551615) >>>> 2017-05-16 10:13:08.941+0000: initiating migration >>>> 2017-05-16T10:17:27.861843Z qemu-kvm: socket_writev_buffer: Got err=32 >>>> for (69776/18446744073709551615) >>>> 2017-05-16 10:18:13.690+0000: initiating migration >>>> 2017-05-16T10:22:32.929689Z qemu-kvm: socket_writev_buffer: Got err=32 >>>> for (69803/18446744073709551615) >>>> 2017-05-16 10:23:19.846+0000: initiating migration >>>> 2017-05-16T10:27:39.175724Z qemu-kvm: socket_writev_buffer: Got err=32 >>>> for (69776/18446744073709551615) >>>> 2017-05-16 10:28:25.141+0000: initiating migration >>>> 2017-05-16T10:28:35.620070Z qemu-kvm: socket_writev_buffer: Got err=32 >>>> for (65753/18446744073709551615) >>>> 2017-05-16 10:29:10.678+0000: initiating migration >>>> 2017-05-16T10:33:29.718527Z qemu-kvm: socket_writev_buffer: Got err=32 >>>> for (53477/18446744073709551615) >>>> 2017-05-16 10:38:35.517+0000: initiating migration >>>> >>> Hi, >>> it seems either qemu issue or misconfiguration. To investigate, we need >>> more data; so could you please share: >>> 1. the domain XML (virtsh -r dumpxml ...) and/or the qemu command line >>> of the affected VM, on the source side >>> 2. the version of QEMU and libvirt that you are running >>> >>> Thanks and bests, >>> >>> -- >>> Francesco Romani >>> Senior SW Eng., Virtualization R&D >>> Red Hat >>> IRC: fromani github: @fromanirh >>> >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users

do you always have the same error?

Are you by any chance using the post copy migration mode?

Could you please share the libvirt debug logs, at least on the source side? https://wiki.libvirt.org/page/DebugLogs You may want to do a test run with the debug logs turned on and disable them just after, those are VERY verbose.

Hi Francesco, > do you always have the same error? Yes

> Are you by any chance using the post copy migration mode? Not sure what that is. The migration is initiated by either putting the host in maintenance, or selecting 'upgrade' from the menu.

> Could you please share the libvirt debug logs, at least on the source side? > > https://wiki.libvirt.org/page/DebugLogs > > You may want to do a test run with the debug logs turned on and disable them > just after, those are VERY verbose. > Before I put it into debug mode, I'd tried to migrate the troublesome VM by itself, and it worked. So I then thought I try putting the host that has that VM into maintenance, and it successfully copied that VM over, but failed on another VM this time (a Linux VM). I took the host out of maintenance, and switched on debug mode (restarting libvirtd made the host rather unhappy, but it sorted itself out), and then tried putting the host back into maintenance. It complained about another VM failing to be migrated, though I couldn't see that VM on the host. It did succeed in moving the VM that was causing it problems, and is now in maintenance mode. So it seems a bit hit and miss. Do you still want the logs?