Hi, Just trying to place in maintenance mode for a version upgrade, and one VM fails to migrate. The other 20-odd move over successfully. In /var/log/libvirt/qemu/, the VM's log on the source reports: 2017-05-16 09:48:06.339+0000: initiating migration 2017-05-16T09:52:25.498932Z qemu-kvm: socket_writev_buffer: Got err=32 for (131328/18446744073709551615) 2017-05-16 09:52:47.311+0000: initiating migration 2017-05-16T09:57:06.755402Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 09:57:55.109+0000: initiating migration 2017-05-16T10:02:14.143221Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:02:59.497+0000: initiating migration 2017-05-16T10:07:18.542872Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:08:03.896+0000: initiating migration 2017-05-16T10:12:23.206731Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:13:08.941+0000: initiating migration 2017-05-16T10:17:27.861843Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:18:13.690+0000: initiating migration 2017-05-16T10:22:32.929689Z qemu-kvm: socket_writev_buffer: Got err=32 for (69803/18446744073709551615) 2017-05-16 10:23:19.846+0000: initiating migration 2017-05-16T10:27:39.175724Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:28:25.141+0000: initiating migration 2017-05-16T10:28:35.620070Z qemu-kvm: socket_writev_buffer: Got err=32 for (65753/18446744073709551615) 2017-05-16 10:29:10.678+0000: initiating migration 2017-05-16T10:33:29.718527Z qemu-kvm: socket_writev_buffer: Got err=32 for (53477/18446744073709551615) 2017-05-16 10:38:35.517+0000: initiating migration On the destination host, it reports: 2017-05-16T10:33:29.598425Z qemu-kvm: Unknown combination of migration flags: 0 2017-05-16T10:33:29.599524Z qemu-kvm: error while loading state section id 2(ram) 2017-05-16T10:33:29.601978Z qemu-kvm: load of migration failed: Invalid argument 2017-05-16 10:33:29.808+0000: shutting down In the engine log: 2017-05-16 11:57:28,675+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [f8aa18b3-97b9-48e2-a681-cf3aaed330a5] VM '4921c5f5-7748-47eb-a90c-8e9ecbd91bcf'(pete_win7) was unexpectedly detected as 'MigratingTo' on VDS '424e6317-ad68-459b-bf88-7292e26710ae'(kvm-ldn-02) (expected on 'e050c27f-8709-404c-b03e-59c0167a824b') It stays in 'preparing for maintenance mode' on the GUI for the host, and reports errors on the status pane below, but then it reports that the host has successfully been put into maintenance mode, even though the VM is still showing as in migration. After more time has elapsed, it reports a failure again, and then eventually the host will again report it is in maintenance mode, and so on. When I hit 'cancel migration' it reports that the migration has been successfully cancelled in the status pane at the bottom, but then shows it still in migration in the upper window. When I select the VM itself, it reports it is "Migrating From: 99%". If I cancel the migration here, it actually does cancel the migration properly. The VM itself is up and running ok. Version of oVirt is 4.1.0.4-1.el7 Thanks in advance for any help. Please let me know if you need any full logs. Regards, Cam
Try involving Francesco from virt team. On Tue, May 16, 2017 at 1:06 PM, cmc <iucounu@gmail.com> wrote:
Hi,
Just trying to place in maintenance mode for a version upgrade, and one VM fails to migrate. The other 20-odd move over successfully. In /var/log/libvirt/qemu/, the VM's log on the source reports:
2017-05-16 09:48:06.339+0000: initiating migration 2017-05-16T09:52:25.498932Z qemu-kvm: socket_writev_buffer: Got err=32 for (131328/18446744073709551615) 2017-05-16 09:52:47.311+0000: initiating migration 2017-05-16T09:57:06.755402Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 09:57:55.109+0000: initiating migration 2017-05-16T10:02:14.143221Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:02:59.497+0000: initiating migration 2017-05-16T10:07:18.542872Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:08:03.896+0000: initiating migration 2017-05-16T10:12:23.206731Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:13:08.941+0000: initiating migration 2017-05-16T10:17:27.861843Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:18:13.690+0000: initiating migration 2017-05-16T10:22:32.929689Z qemu-kvm: socket_writev_buffer: Got err=32 for (69803/18446744073709551615) 2017-05-16 10:23:19.846+0000: initiating migration 2017-05-16T10:27:39.175724Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:28:25.141+0000: initiating migration 2017-05-16T10:28:35.620070Z qemu-kvm: socket_writev_buffer: Got err=32 for (65753/18446744073709551615) 2017-05-16 10:29:10.678+0000: initiating migration 2017-05-16T10:33:29.718527Z qemu-kvm: socket_writev_buffer: Got err=32 for (53477/18446744073709551615) 2017-05-16 10:38:35.517+0000: initiating migration
On the destination host, it reports:
2017-05-16T10:33:29.598425Z qemu-kvm: Unknown combination of migration flags: 0 2017-05-16T10:33:29.599524Z qemu-kvm: error while loading state section id 2(ram) 2017-05-16T10:33:29.601978Z qemu-kvm: load of migration failed: Invalid argument 2017-05-16 10:33:29.808+0000: shutting down
In the engine log:
2017-05-16 11:57:28,675+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [f8aa18b3-97b9-48e2-a681-cf3aaed330a5] VM '4921c5f5-7748-47eb-a90c-8e9ecbd91bcf'(pete_win7) was unexpectedly detected as 'MigratingTo' on VDS '424e6317-ad68-459b-bf88-7292e26710ae'(kvm-ldn-02) (expected on 'e050c27f-8709-404c-b03e-59c0167a824b')
It stays in 'preparing for maintenance mode' on the GUI for the host, and reports errors on the status pane below, but then it reports that the host has successfully been put into maintenance mode, even though the VM is still showing as in migration. After more time has elapsed, it reports a failure again, and then eventually the host will again report it is in maintenance mode, and so on.
When I hit 'cancel migration' it reports that the migration has been successfully cancelled in the status pane at the bottom, but then shows it still in migration in the upper window. When I select the VM itself, it reports it is "Migrating From: 99%". If I cancel the migration here, it actually does cancel the migration properly.
The VM itself is up and running ok.
Version of oVirt is 4.1.0.4-1.el7
Thanks in advance for any help. Please let me know if you need any full logs.
Regards,
Cam _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On 05/16/2017 01:06 PM, cmc wrote:
Hi,
Just trying to place in maintenance mode for a version upgrade, and one VM fails to migrate. The other 20-odd move over successfully. In /var/log/libvirt/qemu/, the VM's log on the source reports:
2017-05-16 09:48:06.339+0000: initiating migration 2017-05-16T09:52:25.498932Z qemu-kvm: socket_writev_buffer: Got err=32 for (131328/18446744073709551615) 2017-05-16 09:52:47.311+0000: initiating migration 2017-05-16T09:57:06.755402Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 09:57:55.109+0000: initiating migration 2017-05-16T10:02:14.143221Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:02:59.497+0000: initiating migration 2017-05-16T10:07:18.542872Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:08:03.896+0000: initiating migration 2017-05-16T10:12:23.206731Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:13:08.941+0000: initiating migration 2017-05-16T10:17:27.861843Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:18:13.690+0000: initiating migration 2017-05-16T10:22:32.929689Z qemu-kvm: socket_writev_buffer: Got err=32 for (69803/18446744073709551615) 2017-05-16 10:23:19.846+0000: initiating migration 2017-05-16T10:27:39.175724Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:28:25.141+0000: initiating migration 2017-05-16T10:28:35.620070Z qemu-kvm: socket_writev_buffer: Got err=32 for (65753/18446744073709551615) 2017-05-16 10:29:10.678+0000: initiating migration 2017-05-16T10:33:29.718527Z qemu-kvm: socket_writev_buffer: Got err=32 for (53477/18446744073709551615) 2017-05-16 10:38:35.517+0000: initiating migration
Hi, it seems either qemu issue or misconfiguration. To investigate, we need more data; so could you please share: 1. the domain XML (virtsh -r dumpxml ...) and/or the qemu command line of the affected VM, on the source side 2. the version of QEMU and libvirt that you are running Thanks and bests, -- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh
Hi Francesco, I left it running after I posted to the list, and it eventually (after many failed attempts) moved the VM without any intervention by me, and then updated the host, so that explains the differences in the versions of qemu between the hosts (they probably would have been the same when I tried the move first). The xml is attached. qemu and libvirt versions on the source host: ipxe-roms-qemu-20160127-5.git6366fa7a.el7.noarch libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64 qemu-img-ev-2.6.0-28.el7_3.9.1.x86_64 qemu-kvm-common-ev-2.6.0-28.el7_3.9.1.x86_64 qemu-kvm-ev-2.6.0-28.el7_3.9.1.x86_64 qemu-kvm-tools-ev-2.6.0-28.el7_3.3.1.x86_64 libvirt-2.0.0-10.el7_3.5.x86_64 libvirt-client-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-config-network-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-config-nwfilter-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-interface-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-lxc-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-network-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-nodedev-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-nwfilter-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-secret-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-storage-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-kvm-2.0.0-10.el7_3.5.x86_64 libvirt-lock-sanlock-2.0.0-10.el7_3.5.x86_64 libvirt-python-2.0.0-2.el7.x86_64 qemu and libvirt versions on the dest host: ipxe-roms-qemu-20160127-5.git6366fa7a.el7.noarch libvirt-daemon-driver-qemu-2.0.0-10.el7_3.4.x86_64 qemu-img-ev-2.6.0-28.el7_3.3.1.x86_64 qemu-kvm-common-ev-2.6.0-28.el7_3.3.1.x86_64 qemu-kvm-ev-2.6.0-28.el7_3.3.1.x86_64 qemu-kvm-tools-ev-2.6.0-28.el7_3.3.1.x86_64 libvirt-client-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-config-nwfilter-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-interface-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-network-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-nodedev-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-nwfilter-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-qemu-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-secret-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-storage-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-kvm-2.0.0-10.el7_3.4.x86_64 libvirt-lock-sanlock-2.0.0-10.el7_3.4.x86_64 libvirt-python-2.0.0-2.el7.x86_64 Thanks, Cam On Wed, May 17, 2017 at 9:12 AM, Francesco Romani <fromani@redhat.com> wrote:
On 05/16/2017 01:06 PM, cmc wrote:
Hi,
Just trying to place in maintenance mode for a version upgrade, and one VM fails to migrate. The other 20-odd move over successfully. In /var/log/libvirt/qemu/, the VM's log on the source reports:
2017-05-16 09:48:06.339+0000: initiating migration 2017-05-16T09:52:25.498932Z qemu-kvm: socket_writev_buffer: Got err=32 for (131328/18446744073709551615) 2017-05-16 09:52:47.311+0000: initiating migration 2017-05-16T09:57:06.755402Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 09:57:55.109+0000: initiating migration 2017-05-16T10:02:14.143221Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:02:59.497+0000: initiating migration 2017-05-16T10:07:18.542872Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:08:03.896+0000: initiating migration 2017-05-16T10:12:23.206731Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:13:08.941+0000: initiating migration 2017-05-16T10:17:27.861843Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:18:13.690+0000: initiating migration 2017-05-16T10:22:32.929689Z qemu-kvm: socket_writev_buffer: Got err=32 for (69803/18446744073709551615) 2017-05-16 10:23:19.846+0000: initiating migration 2017-05-16T10:27:39.175724Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:28:25.141+0000: initiating migration 2017-05-16T10:28:35.620070Z qemu-kvm: socket_writev_buffer: Got err=32 for (65753/18446744073709551615) 2017-05-16 10:29:10.678+0000: initiating migration 2017-05-16T10:33:29.718527Z qemu-kvm: socket_writev_buffer: Got err=32 for (53477/18446744073709551615) 2017-05-16 10:38:35.517+0000: initiating migration
Hi, it seems either qemu issue or misconfiguration. To investigate, we need more data; so could you please share: 1. the domain XML (virtsh -r dumpxml ...) and/or the qemu command line of the affected VM, on the source side 2. the version of QEMU and libvirt that you are running
Thanks and bests,
-- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Just a note on this: a similar thing is now happening with the same VM when I upgrade the other node, i.e., it can't move this one VM over (so far) from one host to another. I will leave it trying overnight to see if it succeeds. Thanks, Cam On Wed, May 17, 2017 at 11:40 AM, cmc <iucounu@gmail.com> wrote:
Hi Francesco,
I left it running after I posted to the list, and it eventually (after many failed attempts) moved the VM without any intervention by me, and then updated the host, so that explains the differences in the versions of qemu between the hosts (they probably would have been the same when I tried the move first). The xml is attached.
qemu and libvirt versions on the source host:
ipxe-roms-qemu-20160127-5.git6366fa7a.el7.noarch libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64 qemu-img-ev-2.6.0-28.el7_3.9.1.x86_64 qemu-kvm-common-ev-2.6.0-28.el7_3.9.1.x86_64 qemu-kvm-ev-2.6.0-28.el7_3.9.1.x86_64 qemu-kvm-tools-ev-2.6.0-28.el7_3.3.1.x86_64
libvirt-2.0.0-10.el7_3.5.x86_64 libvirt-client-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-config-network-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-config-nwfilter-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-interface-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-lxc-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-network-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-nodedev-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-nwfilter-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-secret-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-storage-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-kvm-2.0.0-10.el7_3.5.x86_64 libvirt-lock-sanlock-2.0.0-10.el7_3.5.x86_64 libvirt-python-2.0.0-2.el7.x86_64
qemu and libvirt versions on the dest host:
ipxe-roms-qemu-20160127-5.git6366fa7a.el7.noarch libvirt-daemon-driver-qemu-2.0.0-10.el7_3.4.x86_64 qemu-img-ev-2.6.0-28.el7_3.3.1.x86_64 qemu-kvm-common-ev-2.6.0-28.el7_3.3.1.x86_64 qemu-kvm-ev-2.6.0-28.el7_3.3.1.x86_64 qemu-kvm-tools-ev-2.6.0-28.el7_3.3.1.x86_64
libvirt-client-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-config-nwfilter-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-interface-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-network-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-nodedev-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-nwfilter-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-qemu-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-secret-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-storage-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-kvm-2.0.0-10.el7_3.4.x86_64 libvirt-lock-sanlock-2.0.0-10.el7_3.4.x86_64 libvirt-python-2.0.0-2.el7.x86_64
Thanks,
Cam
On Wed, May 17, 2017 at 9:12 AM, Francesco Romani <fromani@redhat.com> wrote:
On 05/16/2017 01:06 PM, cmc wrote:
Hi,
Just trying to place in maintenance mode for a version upgrade, and one VM fails to migrate. The other 20-odd move over successfully. In /var/log/libvirt/qemu/, the VM's log on the source reports:
2017-05-16 09:48:06.339+0000: initiating migration 2017-05-16T09:52:25.498932Z qemu-kvm: socket_writev_buffer: Got err=32 for (131328/18446744073709551615) 2017-05-16 09:52:47.311+0000: initiating migration 2017-05-16T09:57:06.755402Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 09:57:55.109+0000: initiating migration 2017-05-16T10:02:14.143221Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:02:59.497+0000: initiating migration 2017-05-16T10:07:18.542872Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:08:03.896+0000: initiating migration 2017-05-16T10:12:23.206731Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:13:08.941+0000: initiating migration 2017-05-16T10:17:27.861843Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:18:13.690+0000: initiating migration 2017-05-16T10:22:32.929689Z qemu-kvm: socket_writev_buffer: Got err=32 for (69803/18446744073709551615) 2017-05-16 10:23:19.846+0000: initiating migration 2017-05-16T10:27:39.175724Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:28:25.141+0000: initiating migration 2017-05-16T10:28:35.620070Z qemu-kvm: socket_writev_buffer: Got err=32 for (65753/18446744073709551615) 2017-05-16 10:29:10.678+0000: initiating migration 2017-05-16T10:33:29.718527Z qemu-kvm: socket_writev_buffer: Got err=32 for (53477/18446744073709551615) 2017-05-16 10:38:35.517+0000: initiating migration
Hi, it seems either qemu issue or misconfiguration. To investigate, we need more data; so could you please share: 1. the domain XML (virtsh -r dumpxml ...) and/or the qemu command line of the affected VM, on the source side 2. the version of QEMU and libvirt that you are running
Thanks and bests,
-- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I have to shut the VM down to stop it trying to repeatedly trying to migrate the problematic host (always the same one). If I take it out of maintenance, it will move the VMs back to balance (as per policy), so this is rather inconvenient. It took leaving it overnight and letting oVirt try repeatedly every few minutes to get it to migrate the VM (I can't wait that long, so I've had to shut that VM down for now) On Wed, May 17, 2017 at 4:13 PM, cmc <iucounu@gmail.com> wrote:
Just a note on this: a similar thing is now happening with the same VM when I upgrade the other node, i.e., it can't move this one VM over (so far) from one host to another. I will leave it trying overnight to see if it succeeds.
Thanks,
Cam
On Wed, May 17, 2017 at 11:40 AM, cmc <iucounu@gmail.com> wrote:
Hi Francesco,
I left it running after I posted to the list, and it eventually (after many failed attempts) moved the VM without any intervention by me, and then updated the host, so that explains the differences in the versions of qemu between the hosts (they probably would have been the same when I tried the move first). The xml is attached.
qemu and libvirt versions on the source host:
ipxe-roms-qemu-20160127-5.git6366fa7a.el7.noarch libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64 qemu-img-ev-2.6.0-28.el7_3.9.1.x86_64 qemu-kvm-common-ev-2.6.0-28.el7_3.9.1.x86_64 qemu-kvm-ev-2.6.0-28.el7_3.9.1.x86_64 qemu-kvm-tools-ev-2.6.0-28.el7_3.3.1.x86_64
libvirt-2.0.0-10.el7_3.5.x86_64 libvirt-client-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-config-network-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-config-nwfilter-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-interface-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-lxc-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-network-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-nodedev-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-nwfilter-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-secret-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-storage-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-kvm-2.0.0-10.el7_3.5.x86_64 libvirt-lock-sanlock-2.0.0-10.el7_3.5.x86_64 libvirt-python-2.0.0-2.el7.x86_64
qemu and libvirt versions on the dest host:
ipxe-roms-qemu-20160127-5.git6366fa7a.el7.noarch libvirt-daemon-driver-qemu-2.0.0-10.el7_3.4.x86_64 qemu-img-ev-2.6.0-28.el7_3.3.1.x86_64 qemu-kvm-common-ev-2.6.0-28.el7_3.3.1.x86_64 qemu-kvm-ev-2.6.0-28.el7_3.3.1.x86_64 qemu-kvm-tools-ev-2.6.0-28.el7_3.3.1.x86_64
libvirt-client-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-config-nwfilter-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-interface-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-network-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-nodedev-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-nwfilter-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-qemu-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-secret-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-storage-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-kvm-2.0.0-10.el7_3.4.x86_64 libvirt-lock-sanlock-2.0.0-10.el7_3.4.x86_64 libvirt-python-2.0.0-2.el7.x86_64
Thanks,
Cam
On Wed, May 17, 2017 at 9:12 AM, Francesco Romani <fromani@redhat.com> wrote:
On 05/16/2017 01:06 PM, cmc wrote:
Hi,
Just trying to place in maintenance mode for a version upgrade, and one VM fails to migrate. The other 20-odd move over successfully. In /var/log/libvirt/qemu/, the VM's log on the source reports:
2017-05-16 09:48:06.339+0000: initiating migration 2017-05-16T09:52:25.498932Z qemu-kvm: socket_writev_buffer: Got err=32 for (131328/18446744073709551615) 2017-05-16 09:52:47.311+0000: initiating migration 2017-05-16T09:57:06.755402Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 09:57:55.109+0000: initiating migration 2017-05-16T10:02:14.143221Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:02:59.497+0000: initiating migration 2017-05-16T10:07:18.542872Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:08:03.896+0000: initiating migration 2017-05-16T10:12:23.206731Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:13:08.941+0000: initiating migration 2017-05-16T10:17:27.861843Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:18:13.690+0000: initiating migration 2017-05-16T10:22:32.929689Z qemu-kvm: socket_writev_buffer: Got err=32 for (69803/18446744073709551615) 2017-05-16 10:23:19.846+0000: initiating migration 2017-05-16T10:27:39.175724Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:28:25.141+0000: initiating migration 2017-05-16T10:28:35.620070Z qemu-kvm: socket_writev_buffer: Got err=32 for (65753/18446744073709551615) 2017-05-16 10:29:10.678+0000: initiating migration 2017-05-16T10:33:29.718527Z qemu-kvm: socket_writev_buffer: Got err=32 for (53477/18446744073709551615) 2017-05-16 10:38:35.517+0000: initiating migration
Hi, it seems either qemu issue or misconfiguration. To investigate, we need more data; so could you please share: 1. the domain XML (virtsh -r dumpxml ...) and/or the qemu command line of the affected VM, on the source side 2. the version of QEMU and libvirt that you are running
Thanks and bests,
-- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On 05/18/2017 04:53 PM, cmc wrote:
I have to shut the VM down to stop it trying to repeatedly trying to migrate the problematic host (always the same one). If I take it out of maintenance, it will move the VMs back to balance (as per policy), so this is rather inconvenient. It took leaving it overnight and letting oVirt try repeatedly every few minutes to get it to migrate the VM (I can't wait that long, so I've had to shut that VM down for now)
Hi, do you always have the same error? This: 2017-05-16T10:12:23.206731Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) is most likely found when QEMU is (failing to) transferring the VM state during the migration.
From https://bugzilla.redhat.com/show_bug.cgi?id=1355662 , however, we learn that this message could be a red herring - if we see it, doesn't mean something's wrong.
From oVirt perspective, it seems all good. We need to investigate the lower layers: libvirt, qemu. Let's start.
Are you by any chance using the post copy migration mode? Could you please share the libvirt debug logs, at least on the source side? https://wiki.libvirt.org/page/DebugLogs You may want to do a test run with the debug logs turned on and disable them just after, those are VERY verbose. Thanks,
On Wed, May 17, 2017 at 4:13 PM, cmc <iucounu@gmail.com> wrote:
Just a note on this: a similar thing is now happening with the same VM when I upgrade the other node, i.e., it can't move this one VM over (so far) from one host to another. I will leave it trying overnight to see if it succeeds.
Thanks,
Cam
On Wed, May 17, 2017 at 11:40 AM, cmc <iucounu@gmail.com> wrote:
Hi Francesco,
I left it running after I posted to the list, and it eventually (after many failed attempts) moved the VM without any intervention by me, and then updated the host, so that explains the differences in the versions of qemu between the hosts (they probably would have been the same when I tried the move first). The xml is attached.
qemu and libvirt versions on the source host:
ipxe-roms-qemu-20160127-5.git6366fa7a.el7.noarch libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64 qemu-img-ev-2.6.0-28.el7_3.9.1.x86_64 qemu-kvm-common-ev-2.6.0-28.el7_3.9.1.x86_64 qemu-kvm-ev-2.6.0-28.el7_3.9.1.x86_64 qemu-kvm-tools-ev-2.6.0-28.el7_3.3.1.x86_64
libvirt-2.0.0-10.el7_3.5.x86_64 libvirt-client-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-config-network-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-config-nwfilter-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-interface-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-lxc-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-network-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-nodedev-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-nwfilter-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-secret-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-storage-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-kvm-2.0.0-10.el7_3.5.x86_64 libvirt-lock-sanlock-2.0.0-10.el7_3.5.x86_64 libvirt-python-2.0.0-2.el7.x86_64
qemu and libvirt versions on the dest host:
ipxe-roms-qemu-20160127-5.git6366fa7a.el7.noarch libvirt-daemon-driver-qemu-2.0.0-10.el7_3.4.x86_64 qemu-img-ev-2.6.0-28.el7_3.3.1.x86_64 qemu-kvm-common-ev-2.6.0-28.el7_3.3.1.x86_64 qemu-kvm-ev-2.6.0-28.el7_3.3.1.x86_64 qemu-kvm-tools-ev-2.6.0-28.el7_3.3.1.x86_64
libvirt-client-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-config-nwfilter-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-interface-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-network-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-nodedev-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-nwfilter-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-qemu-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-secret-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-storage-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-kvm-2.0.0-10.el7_3.4.x86_64 libvirt-lock-sanlock-2.0.0-10.el7_3.4.x86_64 libvirt-python-2.0.0-2.el7.x86_64
Thanks,
Cam
On Wed, May 17, 2017 at 9:12 AM, Francesco Romani <fromani@redhat.com> wrote:
On 05/16/2017 01:06 PM, cmc wrote:
Hi,
Just trying to place in maintenance mode for a version upgrade, and one VM fails to migrate. The other 20-odd move over successfully. In /var/log/libvirt/qemu/, the VM's log on the source reports:
2017-05-16 09:48:06.339+0000: initiating migration 2017-05-16T09:52:25.498932Z qemu-kvm: socket_writev_buffer: Got err=32 for (131328/18446744073709551615) 2017-05-16 09:52:47.311+0000: initiating migration 2017-05-16T09:57:06.755402Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 09:57:55.109+0000: initiating migration 2017-05-16T10:02:14.143221Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:02:59.497+0000: initiating migration 2017-05-16T10:07:18.542872Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:08:03.896+0000: initiating migration 2017-05-16T10:12:23.206731Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:13:08.941+0000: initiating migration 2017-05-16T10:17:27.861843Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:18:13.690+0000: initiating migration 2017-05-16T10:22:32.929689Z qemu-kvm: socket_writev_buffer: Got err=32 for (69803/18446744073709551615) 2017-05-16 10:23:19.846+0000: initiating migration 2017-05-16T10:27:39.175724Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:28:25.141+0000: initiating migration 2017-05-16T10:28:35.620070Z qemu-kvm: socket_writev_buffer: Got err=32 for (65753/18446744073709551615) 2017-05-16 10:29:10.678+0000: initiating migration 2017-05-16T10:33:29.718527Z qemu-kvm: socket_writev_buffer: Got err=32 for (53477/18446744073709551615) 2017-05-16 10:38:35.517+0000: initiating migration
Hi, it seems either qemu issue or misconfiguration. To investigate, we need more data; so could you please share: 1. the domain XML (virtsh -r dumpxml ...) and/or the qemu command line of the affected VM, on the source side 2. the version of QEMU and libvirt that you are running
Thanks and bests,
-- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh
Hi Francesco,
do you always have the same error?
Yes
Are you by any chance using the post copy migration mode?
Not sure what that is. The migration is initiated by either putting the host in maintenance, or selecting 'upgrade' from the menu.
Could you please share the libvirt debug logs, at least on the source side?
https://wiki.libvirt.org/page/DebugLogs
You may want to do a test run with the debug logs turned on and disable them just after, those are VERY verbose.
Before I put it into debug mode, I'd tried to migrate the troublesome VM by itself, and it worked. So I then thought I try putting the host that has that VM into maintenance, and it successfully copied that VM over, but failed on another VM this time (a Linux VM). I took the host out of maintenance, and switched on debug mode (restarting libvirtd made the host rather unhappy, but it sorted itself out), and then tried putting the host back into maintenance. It complained about another VM failing to be migrated, though I couldn't see that VM on the host. It did succeed in moving the VM that was causing it problems, and is now in maintenance mode. So it seems a bit hit and miss. Do you still want the logs? Thanks, Cam
On 05/23/2017 04:29 PM, cmc wrote:
Hi Francesco,
do you always have the same error? Yes
Interesting, but this doesn't tell us much
Are you by any chance using the post copy migration mode? Not sure what that is. The migration is initiated by either putting the host in maintenance, or selecting 'upgrade' from the menu.
Postcopy migration is a new live migration mode that performs differently from the old one (called precopy migration). In some scenarios does wonders, in some others... no :) If you don't know about it, it's likely you are using the defaults, so I can check myself (also from the libvirt logs). Please note that from Vdsm perspective there is no difference between user-initiated migration, migrations started because the host goes in maintenance, or any other trigger of migration: they all kickstart the same flow.
Could you please share the libvirt debug logs, at least on the source side?
https://wiki.libvirt.org/page/DebugLogs
You may want to do a test run with the debug logs turned on and disable them just after, those are VERY verbose.
Before I put it into debug mode, I'd tried to migrate the troublesome VM by itself, and it worked. So I then thought I try putting the host that has that VM into maintenance, and it successfully copied that VM over, but failed on another VM this time (a Linux VM). I took the host out of maintenance, and switched on debug mode (restarting libvirtd made the host rather unhappy, but it sorted itself out), and then tried putting the host back into maintenance. It complained about another VM failing to be migrated, though I couldn't see that VM on the host. It did succeed in moving the VM that was causing it problems, and is now in maintenance mode. So it seems a bit hit and miss. Do you still want the logs?
Thanks for your investigations. The issues you are facing definitely point in the direction of low layers of the stack. We may learn something helpful from the libvirt logs, since you already have it could be worth looking at them. So if you want to share them, I'll have a look. -- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh
On 18 May 2017, at 16:53, cmc <iucounu@gmail.com> wrote:
I have to shut the VM down to stop it trying to repeatedly trying to migrate the problematic host (always the same one). If I take it out of maintenance, it will move the VMs back to balance (as per policy), so this is rather inconvenient. It took leaving it overnight and letting oVirt try repeatedly every few minutes to get it to migrate the VM (I can't wait that long, so I've had to shut that VM down for now)
On Wed, May 17, 2017 at 4:13 PM, cmc <iucounu@gmail.com> wrote:
Just a note on this: a similar thing is now happening with the same VM when I upgrade the other node, i.e., it can't move this one VM over (so far) from one host to another. I will leave it trying overnight to see if it succeeds.
is it from older host to newr host or vice versa? can you downgrade the new one to the same version and try again? Thanks, michal
Thanks,
Cam
On Wed, May 17, 2017 at 11:40 AM, cmc <iucounu@gmail.com> wrote:
Hi Francesco,
I left it running after I posted to the list, and it eventually (after many failed attempts) moved the VM without any intervention by me, and then updated the host, so that explains the differences in the versions of qemu between the hosts (they probably would have been the same when I tried the move first). The xml is attached.
qemu and libvirt versions on the source host:
ipxe-roms-qemu-20160127-5.git6366fa7a.el7.noarch libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64 qemu-img-ev-2.6.0-28.el7_3.9.1.x86_64 qemu-kvm-common-ev-2.6.0-28.el7_3.9.1.x86_64 qemu-kvm-ev-2.6.0-28.el7_3.9.1.x86_64 qemu-kvm-tools-ev-2.6.0-28.el7_3.3.1.x86_64
libvirt-2.0.0-10.el7_3.5.x86_64 libvirt-client-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-config-network-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-config-nwfilter-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-interface-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-lxc-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-network-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-nodedev-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-nwfilter-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-secret-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-storage-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-kvm-2.0.0-10.el7_3.5.x86_64 libvirt-lock-sanlock-2.0.0-10.el7_3.5.x86_64 libvirt-python-2.0.0-2.el7.x86_64
qemu and libvirt versions on the dest host:
ipxe-roms-qemu-20160127-5.git6366fa7a.el7.noarch libvirt-daemon-driver-qemu-2.0.0-10.el7_3.4.x86_64 qemu-img-ev-2.6.0-28.el7_3.3.1.x86_64 qemu-kvm-common-ev-2.6.0-28.el7_3.3.1.x86_64 qemu-kvm-ev-2.6.0-28.el7_3.3.1.x86_64 qemu-kvm-tools-ev-2.6.0-28.el7_3.3.1.x86_64
libvirt-client-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-config-nwfilter-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-interface-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-network-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-nodedev-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-nwfilter-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-qemu-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-secret-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-driver-storage-2.0.0-10.el7_3.4.x86_64 libvirt-daemon-kvm-2.0.0-10.el7_3.4.x86_64 libvirt-lock-sanlock-2.0.0-10.el7_3.4.x86_64 libvirt-python-2.0.0-2.el7.x86_64
Thanks,
Cam
On Wed, May 17, 2017 at 9:12 AM, Francesco Romani <fromani@redhat.com> wrote:
On 05/16/2017 01:06 PM, cmc wrote:
Hi,
Just trying to place in maintenance mode for a version upgrade, and one VM fails to migrate. The other 20-odd move over successfully. In /var/log/libvirt/qemu/, the VM's log on the source reports:
2017-05-16 09:48:06.339+0000: initiating migration 2017-05-16T09:52:25.498932Z qemu-kvm: socket_writev_buffer: Got err=32 for (131328/18446744073709551615) 2017-05-16 09:52:47.311+0000: initiating migration 2017-05-16T09:57:06.755402Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 09:57:55.109+0000: initiating migration 2017-05-16T10:02:14.143221Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:02:59.497+0000: initiating migration 2017-05-16T10:07:18.542872Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:08:03.896+0000: initiating migration 2017-05-16T10:12:23.206731Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:13:08.941+0000: initiating migration 2017-05-16T10:17:27.861843Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:18:13.690+0000: initiating migration 2017-05-16T10:22:32.929689Z qemu-kvm: socket_writev_buffer: Got err=32 for (69803/18446744073709551615) 2017-05-16 10:23:19.846+0000: initiating migration 2017-05-16T10:27:39.175724Z qemu-kvm: socket_writev_buffer: Got err=32 for (69776/18446744073709551615) 2017-05-16 10:28:25.141+0000: initiating migration 2017-05-16T10:28:35.620070Z qemu-kvm: socket_writev_buffer: Got err=32 for (65753/18446744073709551615) 2017-05-16 10:29:10.678+0000: initiating migration 2017-05-16T10:33:29.718527Z qemu-kvm: socket_writev_buffer: Got err=32 for (53477/18446744073709551615) 2017-05-16 10:38:35.517+0000: initiating migration
Hi, it seems either qemu issue or misconfiguration. To investigate, we need more data; so could you please share: 1. the domain XML (virtsh -r dumpxml ...) and/or the qemu command line of the affected VM, on the source side 2. the version of QEMU and libvirt that you are running
Thanks and bests,
-- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi Michal,
is it from older host to newr host or vice versa?
They are now on the same version, and I still have some troubles migrating hosts, but not consistently
can you downgrade the new one to the same version and try again?
I've upgraded both now. I'll await Francesco's reply after the latest tests, but as it happens after updating both hosts, I'm guessing it is not specific to the older version Thanks, Cam
participants (4)
-
cmc -
Francesco Romani -
Michal Skrivanek -
Simone Tiraboschi