<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 29 Sep 2016, at 13:59, Davide Ferrari <<a href="mailto:davide@billymob.com" class="">davide@billymob.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class=""><div class="">Hello<br class=""><br class=""></div>Today I've the faulty DIMMs replaced, started the same VM again and did the same migration and this time worked, so it was 100% due to that.<br class=""><br class=""></div>The problem that make me wonder a bit is: if it's the source host with memory problem the one which blocks the correct migration, a faulty DIMM will force you to stop the VMs running on that host, because you cannot simply migrate them away to do the maintenence tasks…</div></div></blockquote><div><br class=""></div>if you have a faulty hw you should do that ASAP as you never know where it is going to affect you. It’s like with disk errors…you may think it’s ok when you rarely write to certain places, but once you try to copy it off the problematic storage and you read every single byte/location you’re screwed…</div><div><br class=""></div><div>Thanks,</div><div>michal</div><div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><br class=""></div><div class="gmail_extra"><br class=""><div class="gmail_quote">2016-09-29 13:53 GMT+02:00 Tomas Jelinek <span dir="ltr" class=""><<a href="mailto:tjelinek@redhat.com" target="_blank" class="">tjelinek@redhat.com</a>></span>:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""><br class="">
<br class="">
----- Original Message -----<br class="">
> From: "Davide Ferrari" <<a href="mailto:davide@billymob.com" class="">davide@billymob.com</a>><br class="">
> To: "users" <<a href="mailto:users@ovirt.org" class="">users@ovirt.org</a>><br class="">
> Sent: Wednesday, September 28, 2016 2:59:59 PM<br class="">
> Subject: [ovirt-users] VM pauses/hangs after migration<br class="">
><br class="">
> Hello<br class="">
><br class="">
> trying to migrate a VM from one host to another, a big VM with 96GB of RAM, I<br class="">
> found that when the migration completes, the VM goes to a paused satte and<br class="">
> cannot be resumed. The libvirt/qemu log it gives is this:<br class="">
><br class="">
> 2016-09-28T12:18:15.679176Z qemu-kvm: error while loading state section id<br class="">
> 2(ram)<br class="">
> 2016-09-28T12:18:15.680010Z qemu-kvm: load of migration failed: Input/output<br class="">
> error<br class="">
> 2016-09-28 12:18:15.872+0000: shutting down<br class="">
> 2016-09-28 12:22:21.467+0000: starting up libvirt version: 1.2.17, package:<br class="">
> 13.el7_2.5 (CentOS BuildSystem < <a href="http://bugs.centos.org/" rel="noreferrer" target="_blank" class="">http://bugs.centos.org</a> >,<br class="">
</span>> <a href="tel:2016-06-23-14" value="+12016062314" class="">2016-06-23-14</a>:23:27, <a href="http://worker1.bsys.centos.org/" rel="noreferrer" target="_blank" class="">worker1.bsys.centos.org</a> ), qemu version: 2.3.0<br class="">
<div class=""><div class="h5">> (qemu-kvm-ev-2.3.0-31.el7.16.<wbr class="">1)<br class="">
> LC_ALL=C PATH=/usr/local/sbin:/usr/<wbr class="">local/bin:/usr/sbin:/usr/bin<br class="">
> QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name <a href="http://front04.billydomain.com/" rel="noreferrer" target="_blank" class="">front04.billydomain.com</a> -S<br class="">
> -machine pc-i440fx-rhel7.2.0,accel=kvm,<wbr class="">usb=off -cpu Haswell-noTSX -m<br class="">
> size=100663296k,slots=16,<wbr class="">maxmem=4294967296k -realtime mlock=off -smp<br class="">
> 32,sockets=16,cores=1,threads=<wbr class="">2 -numa node,nodeid=0,cpus=0-31,mem=<wbr class="">98304<br class="">
> -uuid 4511d1c0-6607-418f-ae75-<wbr class="">34f605b2ad68 -smbios<br class="">
> type=1,manufacturer=oVirt,<wbr class="">product=oVirt<br class="">
> Node,version=7-2.1511.el7.<wbr class="">centos.2.10,serial=4C4C4544-<wbr class="">004A-3310-8054-B2C04F474432,<wbr class="">uuid=4511d1c0-6607-418f-ae75-<wbr class="">34f605b2ad68<br class="">
> -no-user-config -nodefaults -chardev<br class="">
> socket,id=charmonitor,path=/<wbr class="">var/lib/libvirt/qemu/<br class="">
> <a href="http://domain-front04.billydomain.com/monitor.sock,server,nowait" rel="noreferrer" target="_blank" class="">domain-front04.billydomain.<wbr class="">com/monitor.sock,server,nowait</a> -mon<br class="">
> chardev=charmonitor,id=<wbr class="">monitor,mode=control -rtc<br class="">
> base=2016-09-28T14:22:21,<wbr class="">driftfix=slew -global<br class="">
> kvm-pit.lost_tick_policy=<wbr class="">discard -no-hpet -no-shutdown -boot strict=on<br class="">
> -device piix3-usb-uhci,id=usb,bus=pci.<wbr class="">0,addr=0x1.0x2 -device<br class="">
> virtio-scsi-pci,id=scsi0,bus=<wbr class="">pci.0,addr=0x7 -device<br class="">
> virtio-serial-pci,id=virtio-<wbr class="">serial0,max_ports=16,bus=pci.<wbr class="">0,addr=0x4 -drive<br class="">
> if=none,id=drive-ide0-1-0,<wbr class="">readonly=on,format=raw -device<br class="">
> ide-cd,bus=ide.1,unit=0,drive=<wbr class="">drive-ide0-1-0,id=ide0-1-0 -drive<br class="">
> file=/rhev/data-center/<wbr class="">00000001-0001-0001-0001-<wbr class="">0000000003e3/ba2bd397-9222-<wbr class="">424d-aecc-eb652c0169d9/images/<wbr class="">b5b49d5c-2378-4639-9469-<wbr class="">362e37ae7473/24fd0d3c-309b-<wbr class="">458d-9818-4321023afacf,if=<wbr class="">none,id=drive-virtio-disk0,<wbr class="">format=qcow2,serial=b5b49d5c-<wbr class="">2378-4639-9469-362e37ae7473,<wbr class="">cache=none,werror=stop,rerror=<wbr class="">stop,aio=threads<br class="">
> -device<br class="">
> virtio-blk-pci,scsi=off,bus=<wbr class="">pci.0,addr=0x5,drive=drive-<wbr class="">virtio-disk0,id=virtio-disk0,<wbr class="">bootindex=1<br class="">
> -drive<br class="">
> file=/rhev/data-center/<wbr class="">00000001-0001-0001-0001-<wbr class="">0000000003e3/ba2bd397-9222-<wbr class="">424d-aecc-eb652c0169d9/images/<wbr class="">f02ac1ce-52cd-4b81-8b29-<wbr class="">f8006d0469e0/ff4e49c6-3084-<wbr class="">4234-80a1-18a67615c527,if=<wbr class="">none,id=drive-virtio-disk1,<wbr class="">format=raw,serial=f02ac1ce-<wbr class="">52cd-4b81-8b29-f8006d0469e0,<wbr class="">cache=none,werror=stop,rerror=<wbr class="">stop,aio=threads<br class="">
> -device<br class="">
> virtio-blk-pci,scsi=off,bus=<wbr class="">pci.0,addr=0x8,drive=drive-<wbr class="">virtio-disk1,id=virtio-disk1<br class="">
> -netdev tap,fd=30,id=hostnet0,vhost=<wbr class="">on,vhostfd=31 -device<br class="">
> virtio-net-pci,netdev=<wbr class="">hostnet0,id=net0,mac=00:1a:4a:<wbr class="">16:01:56,bus=pci.0,addr=0x3<br class="">
> -chardev<br class="">
> socket,id=charchannel0,path=/<wbr class="">var/lib/libvirt/qemu/channels/<wbr class="">4511d1c0-6607-418f-ae75-<wbr class="">34f605b2ad68.com.redhat.rhevm.<wbr class="">vdsm,server,nowait<br class="">
> -device<br class="">
> virtserialport,bus=virtio-<wbr class="">serial0.0,nr=1,chardev=<wbr class="">charchannel0,id=channel0,name=<wbr class="">com.redhat.rhevm.vdsm<br class="">
> -chardev<br class="">
> socket,id=charchannel1,path=/<wbr class="">var/lib/libvirt/qemu/channels/<wbr class="">4511d1c0-6607-418f-ae75-<wbr class="">34f605b2ad68.org.qemu.guest_<wbr class="">agent.0,server,nowait<br class="">
> -device<br class="">
> virtserialport,bus=virtio-<wbr class="">serial0.0,nr=2,chardev=<wbr class="">charchannel1,id=channel1,name=<wbr class="">org.qemu.guest_agent.0<br class="">
> -chardev spicevmc,id=charchannel2,name=<wbr class="">vdagent -device<br class="">
> virtserialport,bus=virtio-<wbr class="">serial0.0,nr=3,chardev=<wbr class="">charchannel2,id=channel2,name=<wbr class="">com.redhat.spice.0<br class="">
</div></div>> -vnc <a href="http://192.168.10.225:1/" rel="noreferrer" target="_blank" class="">192.168.10.225:1</a> ,password -k es -spice<br class="">
<div class=""><div class="h5">> tls-port=5902,addr=192.168.10.<wbr class="">225,x509-dir=/etc/pki/vdsm/<wbr class="">libvirt-spice,tls-channel=<wbr class="">default,tls-channel=main,tls-<wbr class="">channel=display,tls-channel=<wbr class="">inputs,tls-channel=cursor,tls-<wbr class="">channel=playback,tls-channel=<wbr class="">record,tls-channel=smartcard,<wbr class="">tls-channel=usbredir,seamless-<wbr class="">migration=on<br class="">
> -k es -device<br class="">
> qxl-vga,id=video0,ram_size=<wbr class="">67108864,vram_size=8388608,<wbr class="">vgamem_mb=16,bus=pci.0,addr=<wbr class="">0x2<br class="">
> -incoming tcp: <a href="http://0.0.0.0:49156/" rel="noreferrer" target="_blank" class="">0.0.0.0:49156</a> -device<br class="">
> virtio-balloon-pci,id=<wbr class="">balloon0,bus=pci.0,addr=0x6 -msg timestamp=on<br class="">
> Domain id=5 is tainted: hook-script<br class="">
> red_dispatcher_loadvm_<wbr class="">commands:<br class="">
> KVM: entry failed, hardware error 0x8<br class="">
> RAX=00000000ffffffed RBX=ffff8817ba00c000 RCX=0100000000000000<br class="">
> RDX=0000000000000000<br class="">
> RSI=0000000000000000 RDI=0000000000000046 RBP=ffff8817ba00fe98<br class="">
> RSP=ffff8817ba00fe98<br class="">
> R8 =0000000000000000 R9 =0000000000000000 R10=0000000000000000<br class="">
> R11=0000000000000000<br class="">
> R12=0000000000000006 R13=ffff8817ba00c000 R14=ffff8817ba00c000<br class="">
> R15=0000000000000000<br class="">
> RIP=ffffffff81058e96 RFL=00010286 [--S--P-] CPL=0 II=0 A20=1 SMM=0 HLT=0<br class="">
> ES =0000 0000000000000000 ffffffff 00000000<br class="">
> CS =0010 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]<br class="">
> SS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA]<br class="">
> DS =0000 0000000000000000 ffffffff 00000000<br class="">
> FS =0000 0000000000000000 ffffffff 00000000<br class="">
> GS =0000 ffff8817def80000 ffffffff 00000000<br class="">
> LDT=0000 0000000000000000 ffffffff 00000000<br class="">
> TR =0040 ffff8817def93b80 00002087 00008b00 DPL=0 TSS64-busy<br class="">
> GDT= ffff8817def89000 0000007f<br class="">
> IDT= ffffffffff529000 00000fff<br class="">
> CR0=80050033 CR2=00000000ffffffff CR3=00000017b725b000 CR4=001406e0<br class="">
> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000<br class="">
> DR3=0000000000000000<br class="">
> DR6=00000000ffff0ff0 DR7=0000000000000400<br class="">
> EFER=0000000000000d01<br class="">
> Code=89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f<br class="">
> 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 00 00 00 00 00 55 49<br class="">
> 89 ca<br class="">
> KVM: entry failed, hardware error 0x8<br class="">
> RAX=00000000ffffffed RBX=ffff8817ba008000 RCX=0100000000000000<br class="">
> RDX=0000000000000000<br class="">
> RSI=0000000000000000 RDI=0000000000000046 RBP=ffff8817ba00be98<br class="">
> RSP=ffff8817ba00be98<br class="">
> R8 =0000000000000000 R9 =0000000000000000 R10=0000000000000000<br class="">
> R11=0000000000000000<br class="">
> R12=0000000000000005 R13=ffff8817ba008000 R14=ffff8817ba008000<br class="">
> R15=0000000000000000<br class="">
> RIP=ffffffff81058e96 RFL=00010286 [--S--P-] CPL=0 II=0 A20=1 SMM=0 HLT=0<br class="">
> ES =0000 0000000000000000 ffffffff 00000000<br class="">
> CS =0010 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]<br class="">
> SS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA]<br class="">
> DS =0000 0000000000000000 ffffffff 00000000<br class="">
> FS =0000 0000000000000000 ffffffff 00000000<br class="">
> GS =0000 ffff8817def40000 ffffffff 00000000<br class="">
> LDT=0000 0000000000000000 ffffffff 00000000<br class="">
> TR =0040 ffff8817def53b80 00002087 00008b00 DPL=0 TSS64-busy<br class="">
> GDT= ffff8817def49000 0000007f<br class="">
> IDT= ffffffffff529000 00000fff<br class="">
> CR0=80050033 CR2=00000000ffffffff CR3=00000017b3c9a000 CR4=001406e0<br class="">
> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000<br class="">
> DR3=0000000000000000<br class="">
> DR6=00000000ffff0ff0 DR7=0000000000000400<br class="">
> EFER=0000000000000d01<br class="">
> Code=89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f<br class="">
> 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 00 00 00 00 00 55 49<br class="">
> 89 ca<br class="">
> KVM: entry failed, hardware error 0x80000021<br class="">
><br class="">
> If you're running a guest on an Intel machine without unrestricted mode<br class="">
> support, the failure can be most likely due to the guest entering an invalid<br class="">
> state for Intel VT. For example, the guest maybe running in big real mode<br class="">
> which is not supported on less recent Intel processors.<br class="">
><br class="">
> EAX=ffffffed EBX=ba020000 ECX=00000000 EDX=00000000<br class="">
> ESI=00000000 EDI=00000046 EBP=ba023e98 ESP=ba023e98<br class="">
> EIP=81058e96 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0<br class="">
> ES =0000 00000000 0000ffff 00009300 DPL=0 DS [-WA]<br class="">
> CS =f000 ffff0000 0000ffff 00009b00 DPL=0 CS16 [-RA]<br class="">
> SS =0000 00000000 0000ffff 00009300 DPL=0 DS [-WA]<br class="">
> DS =0000 00000000 0000ffff 00009300 DPL=0 DS [-WA]<br class="">
> FS =0000 00000000 0000ffff 00009300 DPL=0 DS [-WA]<br class="">
> GS =0000 00000000 0000ffff 00009300 DPL=0 DS [-WA]<br class="">
> LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT<br class="">
> TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS64-busy<br class="">
> GDT= 0000000000000000 0000ffff<br class="">
> IDT= 0000000000000000 0000ffff<br class="">
> CR0=80050033 CR2=00007fd826ac20a0 CR3=000000003516c000 CR4=00140060<br class="">
> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000<br class="">
> DR3=0000000000000000<br class="">
> DR6=00000000ffff0ff0 DR7=0000000000000400<br class="">
> EFER=0000000000000d01<br class="">
> Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??> ?? ??<br class="">
> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??<br class="">
> ?? ??<br class="">
><br class="">
><br class="">
> Searching for errors like this I found some bug report about kernel issues<br class="">
> but I don't think it's the case, other VMs spawned from the same image<br class="">
> migrate without any issue. I have toi say that the original host running the<br class="">
> VM has some RAM problem (ECC multibit fault in one DIMM). Maybe that's the<br class="">
> problem?<br class="">
<br class="">
</div></div>that seems quite likely. If you run the same VM on a different host and try to migrate<br class="">
it, does it work?<br class="">
<span class=""><br class="">
> How can I properly read this error log?<br class="">
><br class="">
> Thanks<br class="">
><br class="">
> --<br class="">
> Davide Ferrari<br class="">
> Senior Systems Engineer<br class="">
><br class="">
</span>> ______________________________<wbr class="">_________________<br class="">
> Users mailing list<br class="">
> <a href="mailto:Users@ovirt.org" class="">Users@ovirt.org</a><br class="">
> <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank" class="">http://lists.ovirt.org/<wbr class="">mailman/listinfo/users</a><br class="">
><br class="">
</blockquote></div><br class=""><br clear="all" class=""><br class="">-- <br class=""><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr" class=""><div class="">Davide Ferrari<br class=""></div>Senior Systems Engineer<br class=""></div></div>
</div>
_______________________________________________<br class="">Users mailing list<br class=""><a href="mailto:Users@ovirt.org" class="">Users@ovirt.org</a><br class="">http://lists.ovirt.org/mailman/listinfo/users<br class=""></div></blockquote></div><br class=""></body></html>