[ovirt-users] Details about why a live migration failed

Fri Mar 24 14:18:47 UTC 2017

> On 24 Mar 2017, at 15:15, Davide Ferrari <davide at billymob.com> wrote:
> 
> Mmmh this is all I got from libvirt log on receiver host:
> 
> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name guest=druid-co01,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-5-druid-co01/master-key.aes -machine pc-i440fx-rhel7.2.0,accel=kvm,usb=off -cpu Haswell-noTSX -m size=16777216k,slots=16,maxmem=4294967296k -realtime mlock=off -smp 4,maxcpus=64,sockets=16,cores=4,threads=1 -numa node,nodeid=0,cpus=0-3,mem=16384 -uuid 4f627cc1-9b52-4eef-bf3a-c02e8a6303b8 -smbios 'type=1,manufacturer=oVirt,product=oVirt Node,version=7-2.1511.el7.centos.2.10,serial=4C4C4544-0037-4C10-8031-B7C04F564232,uuid=4f627cc1-9b52-4eef-bf3a-c02e8a6303b8' -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-5-druid-co01.billydoma/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2017-03-24T10:38:03,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x7 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x4 -drive if=none,id=drive-ide0-1-0,readonly=on -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/00000001-0001-0001-0001-0000000003e3/ba2bd397-9222-424d-aecc-eb652c0169d9/images/08b19faa-4b1f-4da8-87a2-2af0700a7906/bdb18a7d-1558-41f9-aa3a-e63407c7881e,format=qcow2,if=none,id=drive-virtio-disk0,serial=08b19faa-4b1f-4da8-87a2-2af0700a7906,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/rhev/data-center/00000001-0001-0001-0001-0000000003e3/ba2bd397-9222-424d-aecc-eb652c0169d9/images/987d84da-188b-45c0-99d0-3dde29ddcb6e/51a1b9ee-b0ae-4208-9806-a319d34db06e,format=qcow2,if=none,id=drive-virtio-disk1,serial=987d84da-188b-45c0-99d0-3dde29ddcb6e,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk1,id=virtio-disk1 -netdev tap,fd=34,id=hostnet0,vhost=on,vhostfd=35 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:16:01:93,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/4f627cc1-9b52-4eef-bf3a-c02e8a6303b8.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/4f627cc1-9b52-4eef-bf3a-c02e8a6303b8.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -spice tls-port=5904,addr=192.168.10.107,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=default,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=8388608,vram64_size_mb=0,vgamem_mb=16,bus=pci.0,addr=0x2 -incoming defer -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on
> Domain id=5 is tainted: hook-script
> 2017-03-24T10:38:03.414396Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
> 2017-03-24T10:38:03.414497Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config
> 2017-03-24 10:41:20.982+0000: shutting down
> 2017-03-24T10:41:20.986633Z qemu-kvm: load of migration failed: Input/output error
> 
> Donating host doesn't say a thing about this VM.
> There's an "input/output error" but I can't see to what is related…

most likely to the migration stream, either the TCP connection was cut short or internal bug
What are the version of qemu on both ends? host OS?

Thanks,
michal

> 
> 
> 
> 
> 
> 2017-03-24 13:38 GMT+01:00 Francesco Romani <fromani at redhat.com <mailto:fromani at redhat.com>>:
> 
> On 03/24/2017 11:58 AM, Davide Ferrari wrote:
> > And this is the vdsm log from vmhost04:
> >
> > Thread-6320717::INFO::2017-03-24
> > 11:41:13,019::migration::712::virt.vm::(monitor_migration)
> > vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::Migration Progress: 190
> > seconds elapsed, 98% of data processed, total data: 16456MB, processed
> > data: 9842MB, remaining data: 386MB, transfer speed 52MBps, zero
> > pages: 1718676MB, compressed: 0MB, dirty rate: -1, memory iteration: -1
> > libvirtEventLoop::DEBUG::2017-03-24
> > 11:41:21,007::vm::4291::virt.vm::(onLibvirtLifecycleEvent)
> > vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::event Suspended detail 0
> > opaque None
> > libvirtEventLoop::INFO::2017-03-24
> > 11:41:21,025::vm::4815::virt.vm::(_logGuestCpuStatus)
> > vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::CPU stopped: onSuspend
> > libvirtEventLoop::DEBUG::2017-03-24
> > 11:41:21,069::vm::4291::virt.vm::(onLibvirtLifecycleEvent)
> > vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::event Resumed detail 0
> > opaque None
> > libvirtEventLoop::INFO::2017-03-24
> > 11:41:21,069::vm::4815::virt.vm::(_logGuestCpuStatus)
> > vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::CPU running: onResume
> > Thread-6320715::DEBUG::2017-03-24
> > 11:41:21,224::migration::715::virt.vm::(stop)
> > vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::stopping migration
> > monitor thread
> > Thread-6320715::ERROR::2017-03-24
> > 11:41:21,225::migration::252::virt.vm::(_recover)
> > vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::operation failed:
> > migration job: unexpectedly failed
> 
> This is surprising (no pun intended)
> With a pretty high chance this comes from libvirt, I'm afraid you need
> to dig in the libvirt logs/journal entries to learn more.
> Vdsm could unfortunately do better than what it is already doing :\
> 
> --
> Francesco Romani
> Senior SW Eng., Virtualization R&D
> Red Hat
> IRC: fromani github: @fromanirh
> 
> _______________________________________________
> Users mailing list
> Users at ovirt.org <mailto:Users at ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users>
> 
> 
> 
> -- 
> Davide Ferrari
> Senior Systems Engineer
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170324/8177084b/attachment-0001.html>