[ovirt-users] Details about why a live migration failed

Fri Mar 24 15:01:53 UTC 2017

Source: CentOS 7.2 - qemu-kvm-ev-2.3.0-31.el7.16.1
Dest: CentOS 7.3 - qemu-kvm-ev-2.6.0-28.el7_3.3.1

To be fair I'm trying to migrate away that VM so I can install updates on
the source host.

2017-03-24 15:18 GMT+01:00 Michal Skrivanek <michal.skrivanek at redhat.com>:

>
> On 24 Mar 2017, at 15:15, Davide Ferrari <davide at billymob.com> wrote:
>
> Mmmh this is all I got from libvirt log on receiver host:
>
> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
> QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name guest=druid-co01,debug-threads=on
> -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/
> qemu/domain-5-druid-co01/master-key.aes -machine
> pc-i440fx-rhel7.2.0,accel=kvm,usb=off -cpu Haswell-noTSX -m
> size=16777216k,slots=16,maxmem=4294967296k -realtime mlock=off -smp
> 4,maxcpus=64,sockets=16,cores=4,threads=1 -numa
> node,nodeid=0,cpus=0-3,mem=16384 -uuid 4f627cc1-9b52-4eef-bf3a-c02e8a6303b8
> -smbios 'type=1,manufacturer=oVirt,product=oVirt
> Node,version=7-2.1511.el7.centos.2.10,serial=4C4C4544-
> 0037-4C10-8031-B7C04F564232,uuid=4f627cc1-9b52-4eef-bf3a-c02e8a6303b8'
> -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/
> var/lib/libvirt/qemu/domain-5-druid-co01.billydoma/monitor.sock,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc
> base=2017-03-24T10:38:03,driftfix=slew -global kvm-pit.lost_tick_policy=discard
> -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
> -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x7 -device
> virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x4
> -drive if=none,id=drive-ide0-1-0,readonly=on -device
> ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive
> file=/rhev/data-center/00000001-0001-0001-0001-0000000003e3/ba2bd397-9222-
> 424d-aecc-eb652c0169d9/images/08b19faa-4b1f-4da8-87a2-
> 2af0700a7906/bdb18a7d-1558-41f9-aa3a-e63407c7881e,format=
> qcow2,if=none,id=drive-virtio-disk0,serial=08b19faa-4b1f-
> 4da8-87a2-2af0700a7906,cache=none,werror=stop,rerror=stop,aio=threads
> -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-
> virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/rhev/data-center/
> 00000001-0001-0001-0001-0000000003e3/ba2bd397-9222-
> 424d-aecc-eb652c0169d9/images/987d84da-188b-45c0-99d0-
> 3dde29ddcb6e/51a1b9ee-b0ae-4208-9806-a319d34db06e,format=
> qcow2,if=none,id=drive-virtio-disk1,serial=987d84da-188b-
> 45c0-99d0-3dde29ddcb6e,cache=none,werror=stop,rerror=stop,aio=threads
> -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk1,id=virtio-disk1
> -netdev tap,fd=34,id=hostnet0,vhost=on,vhostfd=35 -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:16:01:93,bus=pci.0,addr=0x3
> -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/
> 4f627cc1-9b52-4eef-bf3a-c02e8a6303b8.com.redhat.rhevm.vdsm,server,nowait
> -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=
> charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev
> socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/
> 4f627cc1-9b52-4eef-bf3a-c02e8a6303b8.org.qemu.guest_agent.0,server,nowait
> -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=
> charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev
> spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-
> serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0
> -spice tls-port=5904,addr=192.168.10.107,x509-dir=/etc/pki/vdsm/
> libvirt-spice,tls-channel=default,tls-channel=main,tls-
> channel=display,tls-channel=inputs,tls-channel=cursor,tls-
> channel=playback,tls-channel=record,tls-channel=smartcard,
> tls-channel=usbredir,seamless-migration=on -device
> qxl-vga,id=video0,ram_size=67108864,vram_size=8388608,
> vram64_size_mb=0,vgamem_mb=16,bus=pci.0,addr=0x2 -incoming defer -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on
> Domain id=5 is tainted: hook-script
> 2017-03-24T10:38:03.414396Z qemu-kvm: warning: CPU(s) not present in any
> NUMA nodes: 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
> 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51
> 52 53 54 55 56 57 58 59 60 61 62 63
> 2017-03-24T10:38:03.414497Z qemu-kvm: warning: All CPU(s) up to maxcpus
> should be described in NUMA config
> 2017-03-24 10:41:20.982+0000: shutting down
> 2017-03-24T10:41:20.986633Z qemu-kvm: load of migration failed:
> Input/output error
>
> Donating host doesn't say a thing about this VM.
> There's an "input/output error" but I can't see to what is related…
>
>
> most likely to the migration stream, either the TCP connection was cut
> short or internal bug
> What are the version of qemu on both ends? host OS?
>
> Thanks,
> michal
>
>
>
>
>
>
> 2017-03-24 13:38 GMT+01:00 Francesco Romani <fromani at redhat.com>:
>
>>
>> On 03/24/2017 11:58 AM, Davide Ferrari wrote:
>> > And this is the vdsm log from vmhost04:
>> >
>> > Thread-6320717::INFO::2017-03-24
>> > 11:41:13,019::migration::712::virt.vm::(monitor_migration)
>> > vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::Migration Progress: 190
>> > seconds elapsed, 98% of data processed, total data: 16456MB, processed
>> > data: 9842MB, remaining data: 386MB, transfer speed 52MBps, zero
>> > pages: 1718676MB, compressed: 0MB, dirty rate: -1, memory iteration: -1
>> > libvirtEventLoop::DEBUG::2017-03-24
>> > 11:41:21,007::vm::4291::virt.vm::(onLibvirtLifecycleEvent)
>> > vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::event Suspended detail 0
>> > opaque None
>> > libvirtEventLoop::INFO::2017-03-24
>> > 11:41:21,025::vm::4815::virt.vm::(_logGuestCpuStatus)
>> > vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::CPU stopped: onSuspend
>> > libvirtEventLoop::DEBUG::2017-03-24
>> > 11:41:21,069::vm::4291::virt.vm::(onLibvirtLifecycleEvent)
>> > vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::event Resumed detail 0
>> > opaque None
>> > libvirtEventLoop::INFO::2017-03-24
>> > 11:41:21,069::vm::4815::virt.vm::(_logGuestCpuStatus)
>> > vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::CPU running: onResume
>> > Thread-6320715::DEBUG::2017-03-24
>> > 11:41:21,224::migration::715::virt.vm::(stop)
>> > vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::stopping migration
>> > monitor thread
>> > Thread-6320715::ERROR::2017-03-24
>> > 11:41:21,225::migration::252::virt.vm::(_recover)
>> > vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::operation failed:
>> > migration job: unexpectedly failed
>>
>> This is surprising (no pun intended)
>> With a pretty high chance this comes from libvirt, I'm afraid you need
>> to dig in the libvirt logs/journal entries to learn more.
>> Vdsm could unfortunately do better than what it is already doing :\
>>
>> --
>> Francesco Romani
>> Senior SW Eng., Virtualization R&D
>> Red Hat
>> IRC: fromani github: @fromanirh
>>
>> _______________________________________________
>> Users mailing list
>> Users at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>
>
>
> --
> Davide Ferrari
> Senior Systems Engineer
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>

-- 
Davide Ferrari
Senior Systems Engineer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170324/b9d5610a/attachment.html>