[ovirt-devel] Help with issues with migration

[Francesco Romani]

----- Original Message -----
> From: "David Caro" <dcaroest at redhat.com>
> To: devel at ovirt.org
> Sent: Wednesday, February 4, 2015 11:21:39 AM
> Subject: [ovirt-devel] Help with issues with migration
> 
> 
> Hi!
> 
> Upstream phoenix lab has stabilized, but after the outages we are finding
> some
> issues probably caused by it.
> 
> One of them is that th vm migration is not working, ew have two hosts srv05
> and
> srv06, and we want to migrate vms from 06 to 05 but we find those errors on
> vdsm on 05:
> 
> 
>   Feb 04 02:56:50 ovirt-srv05 vdsm[5170]: vdsm vm.Vm WARNING
>   vmId=`43415276-a7bf-4c86-b0e9-70a5f6d39a40`::Unknown type found, device:
>   '{'device': 'unix', 'alias': 'channel0', 'type': 'channel', 'address':
>   {'bus': '0', 'controller': '0', 'type': 'virtio-serial', 'port': '1'}}'
>   found
>   Feb 04 02:56:50 ovirt-srv05 vdsm[5170]: vdsm vm.Vm WARNING
>   vmId=`43415276-a7bf-4c86-b0e9-70a5f6d39a40`::Unknown type found, device:
>   '{'device': 'unix', 'alias': 'channel1', 'type': 'channel', 'address':
>   {'bus': '0', 'controller': '0', 'type': 'virtio-serial', 'port': '2'}}'
>   found
>   Feb 04 02:56:50 ovirt-srv05 vdsm[5170]: vdsm vm.Vm WARNING
>   vmId=`43415276-a7bf-4c86-b0e9-70a5f6d39a40`::Unknown type found, device:
>   '{'device': 'spicevmc', 'alias': 'channel2', 'type': 'channel', 'address':
>   {'bus': '0', 'controller': '0', 'type': 'virtio-serial', 'port': '3'}}'
>   found
>   Feb 04 02:56:50 ovirt-srv05 vdsm[5170]: vdsm vm.Vm ERROR
>   vmId=`43415276-a7bf-4c86-b0e9-70a5f6d39a40`::Alias not found for device
>   type graphics during migration at destination host

All of these are mostly noise. We have BZs basically to silence them, but they
are harmless.

> After that sanlock complains:
>   Feb 04 02:56:50 ovirt-srv05 sanlock[1055]: 2015-02-04 02:56:50-0700 1453
>   [1055]: cmd 9 target pid 6479 not found

There is a BZ for this also, but likely not the root cause here

> but it seems that the vm is starting up:
>   Feb 04 02:56:50 ovirt-srv05 systemd[1]: Starting Virtual Machine
>   qemu-el6-vm03-phx-ovirt-org.
>   -- Subject: Unit machine-qemu\x2del6\x2dvm03\x2dphx\x2dovirt\x2dorg.scope
>   has begun with start-up
>   -- Defined-By: systemd
>   -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
>   --
>   -- Unit machine-qemu\x2del6\x2dvm03\x2dphx\x2dovirt\x2dorg.scope has begun
>   starting up.
>   Feb 04 02:56:50 ovirt-srv05 systemd-machined[5642]: New machine
>   qemu-el6-vm03-phx-ovirt-org.
>   -- Subject: A virtual machine or container has been started
>   -- Defined-By: systemd
>   -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
>   --
>   -- The virtual machine qemu-el6-vm03-phx-ovirt-org with its leader PID 6479
>   has been
>   -- started is now ready to use.
>   Feb 04 02:56:50 ovirt-srv05 systemd[1]: Started Virtual Machine
>   qemu-el6-vm03-phx-ovirt-org.
>   -- Subject: Unit machine-qemu\x2del6\x2dvm03\x2dphx\x2dovirt\x2dorg.scope
>   has finished start-up
>   -- Defined-By: systemd
>   -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
>   --
>   -- Unit machine-qemu\x2del6\x2dvm03\x2dphx\x2dovirt\x2dorg.scope has
>   finished starting up.
>   --
>   -- The start-up result is done.
> 
> 
> But it shuts down:
> Feb 04 02:56:51 ovirt-srv05 systemd-machined[5642]: Machine
> qemu-el6-vm03-phx-ovirt-org terminated.
>   -- Subject: A virtual machine or container has been terminated
>   -- Defined-By: systemd
>   -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
>   --
>   -- The virtual machine qemu-el6-vm03-phx-ovirt-org with its leader PID 6479
>   has been
>   -- shut down.
>   Feb 04 02:56:51 ovirt-srv05 vdsm[5170]: vdsm vm.Vm ERROR
>   vmId=`43415276-a7bf-4c86-b0e9-70a5f6d39a40`::Failed to start a migration
>   destination vm
>                                         Traceback (most recent call last):
>                                           File "/usr/share/vdsm/virt/vm.py",
>                                           line 2298, in _startUnderlyingVm
>                                             self._completeIncomingMigration()
>                                           File "/usr/share/vdsm/virt/vm.py",
>                                           line 4107, in
>                                           _completeIncomingMigration
>                                             self._incomingMigrationFinished.isSet(),
>                                             usedTimeout)
>                                           File "/usr/share/vdsm/virt/vm.py",
>                                           line 4160, in
>                                           _attachLibvirtDomainAfterMigration
>                                             raise
>                                             MigrationError(e.get_error_message())
>                                         MigrationError: Domain not found: no
>                                         domain with matching uuid
>                                         '43415276-a7bf-4c86-b0e9-70a5f6d39a40'
>   Feb 04 02:56:51 ovirt-srv05 vdsm[5170]: vdsm root WARNING File:
>   /var/lib/libvirt/qemu/channels/43415276-a7bf-4c86-b0e9-70a5f6d39a40.com.redhat.rhevm.vdsm
>   already removed
>   Feb 04 02:56:51 ovirt-srv05 vdsm[5170]: vdsm root WARNING File:
>   /var/lib/libvirt/qemu/channels/43415276-a7bf-4c86-b0e9-70a5f6d39a40.org.qemu.guest_agent.0
>   already removed
>   Feb 04 02:56:51 ovirt-srv05 vdsm[5170]: vdsm vm.Vm WARNING
>   vmId=`43415276-a7bf-4c86-b0e9-70a5f6d39a40`::trying to set state to Down
>   when already Down

This just tells us that QEMU failed to run in the dest host. Any more informations
in the QEMU logs and/or related to libvirt?

Bests,

-- 
Francesco Romani
RedHat Engineering Virtualization R & D
Phone: 8261328
IRC: fromani