On 05/18/2017 04:53 PM, cmc wrote:
I have to shut the VM down to stop it trying to repeatedly trying to migrate the problematic host (always the same one). If I take it out of maintenance, it will move the VMs back to balance (as per policy), so this is rather inconvenient. It took leaving it overnight and letting oVirt try repeatedly every few minutes to get it to migrate the VM (I can't wait that long, so I've had to shut that VM down for now)
Hi, do you always have the same error? This: 2017-05-16T10:12:23.206731Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) is most likely found when QEMU is (failing to) transferring the VM state during the migration.
From https://bugzilla.redhat.com/show_bug.cgi?id=1355662 , however, we learn that this message could be a red herring - if we see it, doesn't mean something's wrong.
From oVirt perspective, it seems all good. We need to investigate the lower layers: libvirt, qemu. Let's start.
Are you by any chance using the post copy migration mode? Could you please share the libvirt debug logs, at least on the source side? https://wiki.libvirt.org/page/DebugLogs You may want to do a test run with the debug logs turned on and disable them just after, those are VERY verbose. Thanks,
On Wed, May 17, 2017 at 4:13 PM, cmc <iucounu@gmail.com> wrote:
Just a note on this: a similar thing is now happening with the same VM when I upgrade the other node, i.e., it can't move this one VM over (so far) from one host to another. I will leave it trying overnight to see if it succeeds.
Thanks,
Cam
On Wed, May 17, 2017 at 11:40 AM, cmc <iucounu@gmail.com> wrote:
Hi Francesco,
I left it running after I posted to the list, and it eventually (after many failed attempts) moved the VM without any intervention by me, and then updated the host, so that explains the differences in the versions of qemu between the hosts (they probably would have been the same when I tried the move first). The xml is attached.
qemu and libvirt versions on the source host:
ipxe-roms-qemu-20160127-5.git6366fa7a.el7.noarch libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64 qemu-img-ev-2.6.0-28.el7_3.9.1.x86_64 qemu-kvm-common-ev-2.6.0-28.el7_3.9.1.x86_64 qemu-kvm-ev-2.6.0-28.el7_3.9.1.x86_64 qemu-kvm-tools-ev-2.6.0-28.el7_3.3.1.x86_64
libvirt-2.0.0-10.el7_3.5.x86_64 libvirt-client-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-config-network-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-config-nwfilter-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-interface-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-lxc-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-network-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-nodedev-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-nwfilter-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-secret-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-storage-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-kvm-2.0.0-10.el7_3.5.x86_64 libvirt-lock-sanlock-2.0.0-10.el7_3.5.x86_64 libvirt-python-2.0.0-2.el7.x86_64
qemu and libvirt versions on the dest host:
ipxe-roms-qemu-20160127-5.git6366fa7a.el7.noarch libvirt-daemon-driver-qemu-2.0.0-10.el7_3.4.x86_64 qemu-img-ev-2.6.0-28.el7_3.3.1.x86_64 qemu-kvm-common-ev-2.6.0-28.el7_3.3.1.x86_64 qemu-kvm-ev-2.6.0-28.el7_3.3.1.x86_64 qemu-kvm-tools-ev-2.6.0-28.el7_3.3.1.x86_64
libvirt-client-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-config-nwfilter-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-interface-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-network-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-nodedev-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-nwfilter-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-qemu-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-secret-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-storage-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-kvm-2.0.0-10.el7_3.4.x86_64 libvirt-lock-sanlock-2.0.0-10.el7_3.4.x86_64 libvirt-python-2.0.0-2.el7.x86_64
Thanks,
Cam
On Wed, May 17, 2017 at 9:12 AM, Francesco Romani <fromani@redhat.com> wrote:
On 05/16/2017 01:06 PM, cmc wrote:
Hi,
Just trying to place in maintenance mode for a version upgrade, and one VM fails to migrate. The other 20-odd move over successfully. In /var/log/libvirt/qemu/, the VM's log on the source reports:
2017-05-16 09:48:06.339+0000: initiating migration 2017-05-16T09:52:25.498932Z qemu-kvm: socket_writev_buffer: Got err=32 for (131328/18446744073709551615) 2017-05-16 09:52:47.311+0000: initiating migration 2017-05-16T09:57:06.755402Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 09:57:55.109+0000: initiating migration 2017-05-16T10:02:14.143221Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:02:59.497+0000: initiating migration 2017-05-16T10:07:18.542872Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:08:03.896+0000: initiating migration 2017-05-16T10:12:23.206731Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:13:08.941+0000: initiating migration 2017-05-16T10:17:27.861843Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:18:13.690+0000: initiating migration 2017-05-16T10:22:32.929689Z qemu-kvm: socket_writev_buffer: Got err=32 for (69803/18446744073709551615) 2017-05-16 10:23:19.846+0000: initiating migration 2017-05-16T10:27:39.175724Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:28:25.141+0000: initiating migration 2017-05-16T10:28:35.620070Z qemu-kvm: socket_writev_buffer: Got err=32 for (65753/18446744073709551615) 2017-05-16 10:29:10.678+0000: initiating migration 2017-05-16T10:33:29.718527Z qemu-kvm: socket_writev_buffer: Got err=32 for (53477/18446744073709551615) 2017-05-16 10:38:35.517+0000: initiating migration
Hi, it seems either qemu issue or misconfiguration. To investigate, we need more data; so could you please share: 1. the domain XML (virtsh -r dumpxml ...) and/or the qemu command line of the affected VM, on the source side 2. the version of QEMU and libvirt that you are running
Thanks and bests,
-- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh