Re: [ovirt-users] Details about why a live migration failed

--Apple-Mail=_5B8C2080-30BE-4282-9FAC-08BCCFCC89C1 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 QEMU_AUDIO_DRV=3Dspice /usr/libexec/qemu-kvm -name = guest=3Ddruid-co01,debug-threads=3Don -S -object = secret,id=3DmasterKey0,format=3Draw,file=3D/var/lib/libvirt/qemu/domain-5-= druid-co01/master-key.aes -machine pc-i440fx-rhel7.2.0,accel=3Dkvm,usb=3Do= ff -cpu Haswell-noTSX -m size=3D16777216k,slots=3D16,maxmem=3D4294967296k = -realtime mlock=3Doff -smp 4,maxcpus=3D64,sockets=3D16,cores=3D4,threads=3D= 1 -numa node,nodeid=3D0,cpus=3D0-3,mem=3D16384 -uuid = 4f627cc1-9b52-4eef-bf3a-c02e8a6303b8 -smbios = 'type=3D1,manufacturer=3DoVirt,product=3DoVirt = Node,version=3D7-2.1511.el7.centos.2.10,serial=3D4C4C4544-0037-4C10-8031-B= 7C04F564232,uuid=3D4f627cc1-9b52-4eef-bf3a-c02e8a6303b8' -no-user-config = -nodefaults -chardev = socket,id=3Dcharmonitor,path=3D/var/lib/libvirt/qemu/domain-5-druid-co01.b= illydoma/monitor.sock,server,nowait -mon = chardev=3Dcharmonitor,id=3Dmonitor,mode=3Dcontrol -rtc = base=3D2017-03-24T10:38:03,driftfix=3Dslew -global = kvm-pit.lost_tick_policy=3Ddiscard -no-hpet -no-shutdown -boot strict=3Don= -device piix3-usb-uhci,id=3Dusb,bus=3Dpci.0,addr=3D0x1.0x2 -device = virtio-scsi-pci,id=3Dscsi0,bus=3Dpci.0,addr=3D0x7 -device = virtio-serial-pci,id=3Dvirtio-serial0,max_ports=3D16,bus=3Dpci.0,addr=3D0x= 4 -drive if=3Dnone,id=3Ddrive-ide0-1-0,readonly=3Don -device = ide-cd,bus=3Dide.1,unit=3D0,drive=3Ddrive-ide0-1-0,id=3Dide0-1-0 -drive = file=3D/rhev/data-center/00000001-0001-0001-0001-0000000003e3/ba2bd397-922= 2-424d-aecc-eb652c0169d9/images/08b19faa-4b1f-4da8-87a2-2af0700a7906/bdb18= a7d-1558-41f9-aa3a-e63407c7881e,format=3Dqcow2,if=3Dnone,id=3Ddrive-virtio= -disk0,serial=3D08b19faa-4b1f-4da8-87a2-2af0700a7906,cache=3Dnone,werror=3D= stop,rerror=3Dstop,aio=3Dthreads -device = virtio-blk-pci,scsi=3Doff,bus=3Dpci.0,addr=3D0x5,drive=3Ddrive-virtio-disk= 0,id=3Dvirtio-disk0,bootindex=3D1 -drive = file=3D/rhev/data-center/00000001-0001-0001-0001-0000000003e3/ba2bd397-922= 2-424d-aecc-eb652c0169d9/images/987d84da-188b-45c0-99d0-3dde29ddcb6e/51a1b= 9ee-b0ae-4208-9806-a319d34db06e,format=3Dqcow2,if=3Dnone,id=3Ddrive-virtio= -disk1,serial=3D987d84da-188b-45c0-99d0-3dde29ddcb6e,cache=3Dnone,werror=3D= stop,rerror=3Dstop,aio=3Dthreads -device = virtio-blk-pci,scsi=3Doff,bus=3Dpci.0,addr=3D0x8,drive=3Ddrive-virtio-disk= 1,id=3Dvirtio-disk1 -netdev tap,fd=3D34,id=3Dhostnet0,vhost=3Don,vhostfd=3D= 35 -device = virtio-net-pci,netdev=3Dhostnet0,id=3Dnet0,mac=3D00:1a:4a:16:01:93,bus=3Dp= ci.0,addr=3D0x3 -chardev = socket,id=3Dcharchannel0,path=3D/var/lib/libvirt/qemu/channels/4f627cc1-9b= 52-4eef-bf3a-c02e8a6303b8.com.redhat.rhevm.vdsm,server,nowait -device = virtserialport,bus=3Dvirtio-serial0.0,nr=3D1,chardev=3Dcharchannel0,id=3Dc= hannel0,name=3Dcom.redhat.rhevm.vdsm -chardev = socket,id=3Dcharchannel1,path=3D/var/lib/libvirt/qemu/channels/4f627cc1-9b= 52-4eef-bf3a-c02e8a6303b8.org.qemu.guest_agent.0,server,nowait -device = virtserialport,bus=3Dvirtio-serial0.0,nr=3D2,chardev=3Dcharchannel1,id=3Dc= hannel1,name=3Dorg.qemu.guest_agent.0 -chardev = spicevmc,id=3Dcharchannel2,name=3Dvdagent -device = virtserialport,bus=3Dvirtio-serial0.0,nr=3D3,chardev=3Dcharchannel2,id=3Dc= hannel2,name=3Dcom.redhat.spice.0 -spice = tls-port=3D5904,addr=3D192.168.10.107,x509-dir=3D/etc/pki/vdsm/libvirt-spi= ce,tls-channel=3Ddefault,tls-channel=3Dmain,tls-channel=3Ddisplay,tls-chan= nel=3Dinputs,tls-channel=3Dcursor,tls-channel=3Dplayback,tls-channel=3Drec= ord,tls-channel=3Dsmartcard,tls-channel=3Dusbredir,seamless-migration=3Don= -device = qxl-vga,id=3Dvideo0,ram_size=3D67108864,vram_size=3D8388608,vram64_size_mb= =3D0,vgamem_mb=3D16,bus=3Dpci.0,addr=3D0x2 -incoming defer -device = virtio-balloon-pci,id=3Dballoon0,bus=3Dpci.0,addr=3D0x6 -msg = timestamp=3Don any NUMA nodes: 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 = 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 = 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 maxcpus should be described in NUMA config Input/output error =A6 most likely to the migration stream, either the TCP connection was cut = short or internal bug What are the version of qemu on both ends? host OS? Thanks, michal <mailto:fromani@redhat.com>>: 190 processed -1 detail 0 onSuspend 0 <http://lists.ovirt.org/mailman/listinfo/users> --Apple-Mail=_5B8C2080-30BE-4282-9FAC-08BCCFCC89C1 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><br class=3D""><div><blockquote type=3D"cite" class=3D""><div = class=3D"">On 24 Mar 2017, at 15:15, Davide Ferrari &lt;<a = href=3D"mailto:davide@billymob.com" class=3D&quot;&quot;&gt;davide(a)billymob.com&lt;/a&gt;&amp;gt;= wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><div = dir=3D"ltr" class=3D""><div class=3D""><div class=3D"">Mmmh this is all = I got from libvirt log on receiver host:<br class=3D""><br = class=3D"">LC_ALL=3DC = PATH=3D/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin = QEMU_AUDIO_DRV=3Dspice /usr/libexec/qemu-kvm -name = guest=3Ddruid-co01,debug-threads=3Don -S -object = secret,id=3DmasterKey0,format=3Draw,file=3D/var/lib/libvirt/qemu/domain-5-= druid-co01/master-key.aes -machine pc-i440fx-rhel7.2.0,accel=3Dkvm,usb=3Do= ff -cpu Haswell-noTSX -m size=3D16777216k,slots=3D16,maxmem=3D4294967296k = -realtime mlock=3Doff -smp 4,maxcpus=3D64,sockets=3D16,cores=3D4,threads=3D= 1 -numa node,nodeid=3D0,cpus=3D0-3,mem=3D16384 -uuid = 4f627cc1-9b52-4eef-bf3a-c02e8a6303b8 -smbios = 'type=3D1,manufacturer=3DoVirt,product=3DoVirt = Node,version=3D7-2.1511.el7.centos.2.10,serial=3D4C4C4544-0037-4C10-8031-B= 7C04F564232,uuid=3D4f627cc1-9b52-4eef-bf3a-c02e8a6303b8' -no-user-config = -nodefaults -chardev = socket,id=3Dcharmonitor,path=3D/var/lib/libvirt/qemu/domain-5-druid-co01.b= illydoma/monitor.sock,server,nowait -mon = chardev=3Dcharmonitor,id=3Dmonitor,mode=3Dcontrol -rtc = base=3D2017-03-24T10:38:03,driftfix=3Dslew -global = kvm-pit.lost_tick_policy=3Ddiscard -no-hpet -no-shutdown -boot strict=3Don= -device piix3-usb-uhci,id=3Dusb,bus=3Dpci.0,addr=3D0x1.0x2 -device = virtio-scsi-pci,id=3Dscsi0,bus=3Dpci.0,addr=3D0x7 -device = virtio-serial-pci,id=3Dvirtio-serial0,max_ports=3D16,bus=3Dpci.0,addr=3D0x= 4 -drive if=3Dnone,id=3Ddrive-ide0-1-0,readonly=3Don -device = ide-cd,bus=3Dide.1,unit=3D0,drive=3Ddrive-ide0-1-0,id=3Dide0-1-0 -drive = file=3D/rhev/data-center/00000001-0001-0001-0001-0000000003e3/ba2bd397-922= 2-424d-aecc-eb652c0169d9/images/08b19faa-4b1f-4da8-87a2-2af0700a7906/bdb18= a7d-1558-41f9-aa3a-e63407c7881e,format=3Dqcow2,if=3Dnone,id=3Ddrive-virtio= -disk0,serial=3D08b19faa-4b1f-4da8-87a2-2af0700a7906,cache=3Dnone,werror=3D= stop,rerror=3Dstop,aio=3Dthreads -device = virtio-blk-pci,scsi=3Doff,bus=3Dpci.0,addr=3D0x5,drive=3Ddrive-virtio-disk= 0,id=3Dvirtio-disk0,bootindex=3D1 -drive = file=3D/rhev/data-center/00000001-0001-0001-0001-0000000003e3/ba2bd397-922= 2-424d-aecc-eb652c0169d9/images/987d84da-188b-45c0-99d0-3dde29ddcb6e/51a1b= 9ee-b0ae-4208-9806-a319d34db06e,format=3Dqcow2,if=3Dnone,id=3Ddrive-virtio= -disk1,serial=3D987d84da-188b-45c0-99d0-3dde29ddcb6e,cache=3Dnone,werror=3D= stop,rerror=3Dstop,aio=3Dthreads -device = virtio-blk-pci,scsi=3Doff,bus=3Dpci.0,addr=3D0x8,drive=3Ddrive-virtio-disk= 1,id=3Dvirtio-disk1 -netdev tap,fd=3D34,id=3Dhostnet0,vhost=3Don,vhostfd=3D= 35 -device = virtio-net-pci,netdev=3Dhostnet0,id=3Dnet0,mac=3D00:1a:4a:16:01:93,bus=3Dp= ci.0,addr=3D0x3 -chardev = socket,id=3Dcharchannel0,path=3D/var/lib/libvirt/qemu/channels/4f627cc1-9b= 52-4eef-bf3a-c02e8a6303b8.com.redhat.rhevm.vdsm,server,nowait -device = virtserialport,bus=3Dvirtio-serial0.0,nr=3D1,chardev=3Dcharchannel0,id=3Dc= hannel0,name=3Dcom.redhat.rhevm.vdsm -chardev = socket,id=3Dcharchannel1,path=3D/var/lib/libvirt/qemu/channels/4f627cc1-9b= 52-4eef-bf3a-c02e8a6303b8.org.qemu.guest_agent.0,server,nowait -device = virtserialport,bus=3Dvirtio-serial0.0,nr=3D2,chardev=3Dcharchannel1,id=3Dc= hannel1,name=3Dorg.qemu.guest_agent.0 -chardev = spicevmc,id=3Dcharchannel2,name=3Dvdagent -device = virtserialport,bus=3Dvirtio-serial0.0,nr=3D3,chardev=3Dcharchannel2,id=3Dc= hannel2,name=3Dcom.redhat.spice.0 -spice = tls-port=3D5904,addr=3D192.168.10.107,x509-dir=3D/etc/pki/vdsm/libvirt-spi= ce,tls-channel=3Ddefault,tls-channel=3Dmain,tls-channel=3Ddisplay,tls-chan= nel=3Dinputs,tls-channel=3Dcursor,tls-channel=3Dplayback,tls-channel=3Drec= ord,tls-channel=3Dsmartcard,tls-channel=3Dusbredir,seamless-migration=3Don= -device = qxl-vga,id=3Dvideo0,ram_size=3D67108864,vram_size=3D8388608,vram64_size_mb= =3D0,vgamem_mb=3D16,bus=3Dpci.0,addr=3D0x2 -incoming defer -device = virtio-balloon-pci,id=3Dballoon0,bus=3Dpci.0,addr=3D0x6 -msg = timestamp=3Don<br class=3D"">Domain id=3D5 is tainted: hook-script<br = class=3D"">2017-03-24T10:38:03.414396Z qemu-kvm: warning: CPU(s) not = present in any NUMA nodes: 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 = 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 = 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63<br = class=3D"">2017-03-24T10:38:03.414497Z qemu-kvm: warning: All CPU(s) up = to maxcpus should be described in NUMA config<br class=3D"">2017-03-24 = 10:41:20.982+0000: shutting down<br class=3D"">2017-03-24T10:41:20.986633Z= qemu-kvm: load of migration failed: Input/output error<br class=3D""><br = class=3D""></div>Donating host doesn't say a thing about this VM.<br = class=3D""></div>There's an "input/output error" but I can't see to what = is related=E2=80=A6</div></div></blockquote><div><br class=3D""></div>most= likely to the migration stream, either the TCP connection was cut short = or internal bug</div><div>What are the version of qemu on both ends? = host OS?</div><div><br = class=3D""></div><div>Thanks,</div><div>michal</div><div><br = class=3D""><blockquote type=3D"cite" class=3D""><div class=3D""><div = dir=3D"ltr" class=3D""><br class=3D""><br class=3D""><div class=3D""><div = class=3D""><br class=3D""><br class=3D""></div></div></div><div = class=3D"gmail_extra"><br class=3D""><div class=3D"gmail_quote">2017-03-24= 13:38 GMT+01:00 Francesco Romani <span dir=3D"ltr" class=3D"">&lt;<a = href=3D"mailto:fromani@redhat.com" target=3D"_blank" = class=3D"">fromani@redhat.com</a>&gt;</span>:<br class=3D""><blockquote = class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc = solid;padding-left:1ex"><span class=3D""><br class=3D""> On 03/24/2017 11:58 AM, Davide Ferrari wrote:<br class=3D""> &gt; And this is the vdsm log from vmhost04:<br class=3D""> &gt;<br class=3D""> </span><div class=3D""><div class=3D"h5">&gt; = Thread-6320717::INFO::2017-03-<wbr class=3D"">24<br class=3D""> &gt; 11:41:13,019::migration::712::<wbr = class=3D"">virt.vm::(monitor_migration)<br class=3D""> &gt; vmId=3D`4f627cc1-9b52-4eef-bf3a-<wbr = class=3D"">c02e8a6303b8`::Migration Progress: 190<br class=3D""> &gt; seconds elapsed, 98% of data processed, total data: 16456MB, = processed<br class=3D""> &gt; data: 9842MB, remaining data: 386MB, transfer speed 52MBps, zero<br = class=3D""> &gt; pages: 1718676MB, compressed: 0MB, dirty rate: -1, memory = iteration: -1<br class=3D""> &gt; libvirtEventLoop::DEBUG::2017-<wbr class=3D"">03-24<br class=3D""> &gt; 11:41:21,007::vm::4291::virt.<wbr = class=3D"">vm::(onLibvirtLifecycleEvent)<br class=3D""> &gt; vmId=3D`4f627cc1-9b52-4eef-bf3a-<wbr class=3D"">c02e8a6303b8`::event = Suspended detail 0<br class=3D""> &gt; opaque None<br class=3D""> &gt; libvirtEventLoop::INFO::2017-<wbr class=3D"">03-24<br class=3D""> &gt; 11:41:21,025::vm::4815::virt.<wbr = class=3D"">vm::(_logGuestCpuStatus)<br class=3D""> &gt; vmId=3D`4f627cc1-9b52-4eef-bf3a-<wbr class=3D"">c02e8a6303b8`::CPU = stopped: onSuspend<br class=3D""> &gt; libvirtEventLoop::DEBUG::2017-<wbr class=3D"">03-24<br class=3D""> &gt; 11:41:21,069::vm::4291::virt.<wbr = class=3D"">vm::(onLibvirtLifecycleEvent)<br class=3D""> &gt; vmId=3D`4f627cc1-9b52-4eef-bf3a-<wbr class=3D"">c02e8a6303b8`::event = Resumed detail 0<br class=3D""> &gt; opaque None<br class=3D""> &gt; libvirtEventLoop::INFO::2017-<wbr class=3D"">03-24<br class=3D""> &gt; 11:41:21,069::vm::4815::virt.<wbr = class=3D"">vm::(_logGuestCpuStatus)<br class=3D""> &gt; vmId=3D`4f627cc1-9b52-4eef-bf3a-<wbr class=3D"">c02e8a6303b8`::CPU = running: onResume<br class=3D""> &gt; Thread-6320715::DEBUG::2017-<wbr class=3D"">03-24<br class=3D""> &gt; 11:41:21,224::migration::715::<wbr class=3D"">virt.vm::(stop)<br = class=3D""> &gt; vmId=3D`4f627cc1-9b52-4eef-bf3a-<wbr = class=3D"">c02e8a6303b8`::stopping migration<br class=3D""> &gt; monitor thread<br class=3D""> &gt; Thread-6320715::ERROR::2017-<wbr class=3D"">03-24<br class=3D""> &gt; 11:41:21,225::migration::252::<wbr class=3D"">virt.vm::(_recover)<br = class=3D""> &gt; vmId=3D`4f627cc1-9b52-4eef-bf3a-<wbr = class=3D"">c02e8a6303b8`::operation failed:<br class=3D""> &gt; migration job: unexpectedly failed<br class=3D""> <br class=3D""> </div></div>This is surprising (no pun intended)<br class=3D""> With a pretty high chance this comes from libvirt, I'm afraid you = need<br class=3D""> to dig in the libvirt logs/journal entries to learn more.<br class=3D""> Vdsm could unfortunately do better than what it is already doing :\<br = class=3D""> <span class=3D"HOEnZb"><font color=3D"#888888" class=3D""><br class=3D""> --<br class=3D""> Francesco Romani<br class=3D""> Senior SW Eng., Virtualization R&amp;D<br class=3D""> Red Hat<br class=3D""> IRC: fromani github: @fromanirh<br class=3D""> <br class=3D""> ______________________________<wbr class=3D"">_________________<br = class=3D""> Users mailing list<br class=3D""> <a href=3D"mailto:Users@ovirt.org" class=3D&quot;&quot;&gt;Users(a)ovirt.org&lt;/a&gt;&lt;br = class=3D""> <a href=3D"http://lists.ovirt.org/mailman/listinfo/users" = rel=3D"noreferrer" target=3D"_blank" = class=3D"">http://lists.ovirt.org/<wbr = class=3D"">mailman/listinfo/users</a><br class=3D""> </font></span></blockquote></div><br class=3D""><br clear=3D"all" = class=3D""><br class=3D"">-- <br class=3D""><div class=3D"gmail_signature"= data-smartmail=3D"gmail_signature"><div dir=3D"ltr" class=3D""><div = class=3D"">Davide Ferrari<br class=3D""></div>Senior Systems Engineer<br = class=3D""></div></div> </div> _______________________________________________<br class=3D"">Users = mailing list<br class=3D""><a href=3D"mailto:Users@ovirt.org" = class=3D&quot;&quot;&gt;Users(a)ovirt.org&lt;/a&gt;&lt;br = class=3D"">http://lists.ovirt.org/mailman/listinfo/users<br = class=3D""></div></blockquote></div><br class=3D""></body></html>= --Apple-Mail=_5B8C2080-30BE-4282-9FAC-08BCCFCC89C1--