[ovirt-users] migration of a VM: fails

Mon May 22 06:37:03 UTC 2017

On 05/18/2017 04:53 PM, cmc wrote:
> I have to shut the VM down to stop it trying to repeatedly trying to
> migrate the problematic host (always the same one). If I take it out
> of maintenance, it will move the VMs back to balance (as per policy),
> so this is rather inconvenient. It took leaving it overnight and
> letting oVirt try repeatedly every few minutes to get it to migrate
> the VM (I can't wait that long, so I've had to shut that VM down for
> now)

Hi,

do you always have the same error?

This:

2017-05-16T10:12:23.206731Z qemu-kvm: socket_writev_buffer: Got err=32
for (69776/18446744073709551615)

is most likely found when QEMU is (failing to) transferring the VM state
during the migration.

>From https://bugzilla.redhat.com/show_bug.cgi?id=1355662 , however, we
learn that this message could be a red herring - if we see it, doesn't
mean something's wrong.

>From oVirt perspective, it seems all good. We need to investigate the
lower layers: libvirt, qemu. Let's start.

Are you by any chance using the post copy migration mode?

Could you please share the libvirt debug logs, at least on the source side?

https://wiki.libvirt.org/page/DebugLogs

You may want to do a test run with the debug logs turned on and disable them
just after, those are VERY verbose.

Thanks,

>
> On Wed, May 17, 2017 at 4:13 PM, cmc <iucounu at gmail.com> wrote:
>> Just a note on this: a similar thing is now happening with the same VM
>> when I upgrade the other node, i.e., it can't move this one VM over
>> (so far) from one host to another. I will leave it trying overnight to
>> see if it succeeds.
>>
>> Thanks,
>>
>> Cam
>>
>> On Wed, May 17, 2017 at 11:40 AM, cmc <iucounu at gmail.com> wrote:
>>> Hi Francesco,
>>>
>>> I left it running after I posted to the list, and it eventually (after
>>> many failed attempts) moved the VM without any intervention by me, and
>>> then updated the host, so that explains the differences in the
>>> versions of qemu between the hosts (they probably would have been the
>>> same when I tried the move first). The xml is attached.
>>>
>>> qemu and libvirt versions on the source host:
>>>
>>> ipxe-roms-qemu-20160127-5.git6366fa7a.el7.noarch
>>> libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64
>>> qemu-img-ev-2.6.0-28.el7_3.9.1.x86_64
>>> qemu-kvm-common-ev-2.6.0-28.el7_3.9.1.x86_64
>>> qemu-kvm-ev-2.6.0-28.el7_3.9.1.x86_64
>>> qemu-kvm-tools-ev-2.6.0-28.el7_3.3.1.x86_64
>>>
>>> libvirt-2.0.0-10.el7_3.5.x86_64
>>> libvirt-client-2.0.0-10.el7_3.5.x86_64
>>> libvirt-daemon-2.0.0-10.el7_3.5.x86_64
>>> libvirt-daemon-config-network-2.0.0-10.el7_3.5.x86_64
>>> libvirt-daemon-config-nwfilter-2.0.0-10.el7_3.5.x86_64
>>> libvirt-daemon-driver-interface-2.0.0-10.el7_3.5.x86_64
>>> libvirt-daemon-driver-lxc-2.0.0-10.el7_3.5.x86_64
>>> libvirt-daemon-driver-network-2.0.0-10.el7_3.5.x86_64
>>> libvirt-daemon-driver-nodedev-2.0.0-10.el7_3.5.x86_64
>>> libvirt-daemon-driver-nwfilter-2.0.0-10.el7_3.5.x86_64
>>> libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64
>>> libvirt-daemon-driver-secret-2.0.0-10.el7_3.5.x86_64
>>> libvirt-daemon-driver-storage-2.0.0-10.el7_3.5.x86_64
>>> libvirt-daemon-kvm-2.0.0-10.el7_3.5.x86_64
>>> libvirt-lock-sanlock-2.0.0-10.el7_3.5.x86_64
>>> libvirt-python-2.0.0-2.el7.x86_64
>>>
>>> qemu and libvirt versions on the dest host:
>>>
>>> ipxe-roms-qemu-20160127-5.git6366fa7a.el7.noarch
>>> libvirt-daemon-driver-qemu-2.0.0-10.el7_3.4.x86_64
>>> qemu-img-ev-2.6.0-28.el7_3.3.1.x86_64
>>> qemu-kvm-common-ev-2.6.0-28.el7_3.3.1.x86_64
>>> qemu-kvm-ev-2.6.0-28.el7_3.3.1.x86_64
>>> qemu-kvm-tools-ev-2.6.0-28.el7_3.3.1.x86_64
>>>
>>> libvirt-client-2.0.0-10.el7_3.4.x86_64
>>> libvirt-daemon-2.0.0-10.el7_3.4.x86_64
>>> libvirt-daemon-config-nwfilter-2.0.0-10.el7_3.4.x86_64
>>> libvirt-daemon-driver-interface-2.0.0-10.el7_3.4.x86_64
>>> libvirt-daemon-driver-network-2.0.0-10.el7_3.4.x86_64
>>> libvirt-daemon-driver-nodedev-2.0.0-10.el7_3.4.x86_64
>>> libvirt-daemon-driver-nwfilter-2.0.0-10.el7_3.4.x86_64
>>> libvirt-daemon-driver-qemu-2.0.0-10.el7_3.4.x86_64
>>> libvirt-daemon-driver-secret-2.0.0-10.el7_3.4.x86_64
>>> libvirt-daemon-driver-storage-2.0.0-10.el7_3.4.x86_64
>>> libvirt-daemon-kvm-2.0.0-10.el7_3.4.x86_64
>>> libvirt-lock-sanlock-2.0.0-10.el7_3.4.x86_64
>>> libvirt-python-2.0.0-2.el7.x86_64
>>>
>>>
>>> Thanks,
>>>
>>> Cam
>>>
>>> On Wed, May 17, 2017 at 9:12 AM, Francesco Romani <fromani at redhat.com> wrote:
>>>> On 05/16/2017 01:06 PM, cmc wrote:
>>>>> Hi,
>>>>>
>>>>> Just trying to place in maintenance mode for a version upgrade, and
>>>>> one VM fails to migrate. The other 20-odd move over successfully. In
>>>>> /var/log/libvirt/qemu/, the VM's log on the source reports:
>>>>>
>>>>> 2017-05-16 09:48:06.339+0000: initiating migration
>>>>> 2017-05-16T09:52:25.498932Z qemu-kvm: socket_writev_buffer: Got err=32
>>>>> for (131328/18446744073709551615)
>>>>> 2017-05-16 09:52:47.311+0000: initiating migration
>>>>> 2017-05-16T09:57:06.755402Z qemu-kvm: socket_writev_buffer: Got err=32
>>>>> for (69776/18446744073709551615)
>>>>> 2017-05-16 09:57:55.109+0000: initiating migration
>>>>> 2017-05-16T10:02:14.143221Z qemu-kvm: socket_writev_buffer: Got err=32
>>>>> for (69776/18446744073709551615)
>>>>> 2017-05-16 10:02:59.497+0000: initiating migration
>>>>> 2017-05-16T10:07:18.542872Z qemu-kvm: socket_writev_buffer: Got err=32
>>>>> for (69776/18446744073709551615)
>>>>> 2017-05-16 10:08:03.896+0000: initiating migration
>>>>> 2017-05-16T10:12:23.206731Z qemu-kvm: socket_writev_buffer: Got err=32
>>>>> for (69776/18446744073709551615)
>>>>> 2017-05-16 10:13:08.941+0000: initiating migration
>>>>> 2017-05-16T10:17:27.861843Z qemu-kvm: socket_writev_buffer: Got err=32
>>>>> for (69776/18446744073709551615)
>>>>> 2017-05-16 10:18:13.690+0000: initiating migration
>>>>> 2017-05-16T10:22:32.929689Z qemu-kvm: socket_writev_buffer: Got err=32
>>>>> for (69803/18446744073709551615)
>>>>> 2017-05-16 10:23:19.846+0000: initiating migration
>>>>> 2017-05-16T10:27:39.175724Z qemu-kvm: socket_writev_buffer: Got err=32
>>>>> for (69776/18446744073709551615)
>>>>> 2017-05-16 10:28:25.141+0000: initiating migration
>>>>> 2017-05-16T10:28:35.620070Z qemu-kvm: socket_writev_buffer: Got err=32
>>>>> for (65753/18446744073709551615)
>>>>> 2017-05-16 10:29:10.678+0000: initiating migration
>>>>> 2017-05-16T10:33:29.718527Z qemu-kvm: socket_writev_buffer: Got err=32
>>>>> for (53477/18446744073709551615)
>>>>> 2017-05-16 10:38:35.517+0000: initiating migration
>>>>>
>>>> Hi,
>>>> it seems either qemu issue or misconfiguration. To investigate, we need
>>>> more data; so could you please share:
>>>> 1. the domain XML (virtsh -r dumpxml ...) and/or the qemu command line
>>>> of the affected VM, on the source side
>>>> 2. the version of QEMU and libvirt that you are running
>>>>
>>>> Thanks and bests,
>>>>
>>>> --
>>>> Francesco Romani
>>>> Senior SW Eng., Virtualization R&D
>>>> Red Hat
>>>> IRC: fromani github: @fromanirh
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users at ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/users

-- 
Francesco Romani
Senior SW Eng., Virtualization R&D
Red Hat
IRC: fromani github: @fromanirh