<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi,<div><br></div><div><div>Wondering if anyone has encountered the same issue as me. On one host in my cluster, if I migrate in a guest, I cannot migrate it out to another host? The get "stuck" there, so to speak. Same when a guest is started on that particular host, it is impossible to migrate them out again.</div><div><br></div><div>iptables are flushed, for excluding purposes. vdsmd, libvirtd and sanlock are all running.</div><div><br></div><div>This is what I caught from libvirtd.log:</div><div>2012-08-02 11:44:07.542+0000: 4231: debug : qemuMonitorUnref:248 : QEMU_MONITOR_UNREF: mon=0x7fe7e40c8410 refs=3</div><div><div>2012-08-02 11:44:07.542+0000: 4232: debug : qemuMonitorUnref:248 : QEMU_MONITOR_UNREF: mon=0x7fe7e40c8410 refs=2</div><div>2012-08-02 11:44:07.542+0000: 4232: debug : virConnectIsAlive:18395 : conn=0x7fe7e00c4e50</div><div>2012-08-02 11:44:07.562+0000: 4236: debug : virDomainMigrateSetMaxDowntime:16565 : dom=0x7fe7d8000cc0, (VM: name=milli, uuid=2291f0d8-6341-4821-8774-2ed84b8bccc1), downtime=50, flags=0</div><div>2012-08-02 11:44:07.562+0000: 4236: debug : qemuDomainObjBeginJobInternal:753 : Starting job: migration operation (async=migration out)</div><div>2012-08-02 11:44:07.562+0000: 4236: debug : qemuDomainMigrateSetMaxDowntime:9500 : Setting migration downtime to 50ms</div><div>2012-08-02 11:44:07.562+0000: 4236: debug : qemuMonitorRef:239 : QEMU_MONITOR_REF: mon=0x7fe7e40c8410 refs=3</div><div>2012-08-02 11:44:07.562+0000: 4236: debug : qemuMonitorSetMigrationDowntime:1753 : mon=0x7fe7e40c8410 downtime=50</div><div>2012-08-02 11:44:07.562+0000: 4236: debug : qemuMonitorSend:861 : QEMU_MONITOR_SEND_MSG: mon=0x7fe7e40c8410 msg={"execute":"migrate_set_downtime","arguments":{"value":0,050000},"id":"libvirt-261"}</div><div> fd=-1</div><div>2012-08-02 11:44:07.563+0000: 4231: debug : qemuMonitorRef:239 : QEMU_MONITOR_REF: mon=0x7fe7e40c8410 refs=4</div><div>2012-08-02 11:44:07.563+0000: 4231: debug : qemuMonitorIOWrite:470 : QEMU_MONITOR_IO_WRITE: mon=0x7fe7e40c8410 buf={"execute":"migrate_set_downtime","arguments":{"value":0,050000},"id":"libvirt-261"}</div><div> len=86 ret=86 errno=11</div><div>2012-08-02 11:44:07.563+0000: 4231: debug : qemuMonitorUnref:248 : QEMU_MONITOR_UNREF: mon=0x7fe7e40c8410 refs=3</div><div>2012-08-02 11:44:07.564+0000: 4231: debug : qemuMonitorRef:239 : QEMU_MONITOR_REF: mon=0x7fe7e40c8410 refs=4</div><div>2012-08-02 11:44:07.564+0000: 4231: debug : qemuMonitorIOProcess:365 : QEMU_MONITOR_IO_PROCESS: mon=0x7fe7e40c8410 buf={"error": {"class": "JSONParsing", "desc": "Invalid JSON syntax", "data": {}}}</div><div> len=80</div><div>2012-08-02 11:44:07.564+0000: 4231: debug : qemuMonitorUnref:248 : QEMU_MONITOR_UNREF: mon=0x7fe7e40c8410 refs=3</div><div>2012-08-02 11:44:07.564+0000: 4236: error : qemuMonitorJSONCheckError:331 : internal error unable to execute QEMU command 'migrate_set_downtime': Invalid JSON syntax</div><div>2012-08-02 11:44:07.564+0000: 4236: debug : qemuMonitorUnref:248 : QEMU_MONITOR_UNREF: mon=0x7fe7e40c8410 refs=2</div><div>2012-08-02 11:44:07.564+0000: 4236: debug : qemuDomainObjEndJob:870 : Stopping job: migration operation (async=migration out)</div><div>2012-08-02 11:44:07.564+0000: 4231: debug : qemuMonitorRef:239 : QEMU_MONITOR_REF: mon=0x7fe7e40c8410 refs=3</div><div>2012-08-02 11:44:07.564+0000: 4231: debug : qemuMonitorIOProcess:365 : QEMU_MONITOR_IO_PROCESS: mon=0x7fe7e40c8410 buf={"error": {"class": "JSONParsing", "desc": "Invalid JSON syntax", "data": {}}}</div><div> len=80</div><div>2012-08-02 11:44:07.564+0000: 4236: debug : virDomainFree:2345 : dom=0x7fe7d8000cc0, (VM: name=milli, uuid=2291f0d8-6341-4821-8774-2ed84b8bccc1)</div><div>2012-08-02 11:44:07.564+0000: 4231: error : qemuMonitorJSONIOProcessLine:156 : internal error Unexpected JSON reply '{"error": {"class": "JSONParsing", "desc": "Invalid JSON syntax", "data": {}}}'</div><div>2012-08-02 11:44:07.564+0000: 4231: debug : qemuMonitorIO:645 : Error on monitor internal error Unexpected JSON reply '{"error": {"class": "JSONParsing", "desc": "Invalid JSON syntax", "data": {}}}'</div><div>2012-08-02 11:44:07.564+0000: 4231: debug : qemuMonitorUnref:248 : QEMU_MONITOR_UNREF: mon=0x7fe7e40c8410 refs=2</div><div>2012-08-02 11:44:07.564+0000: 4231: debug : qemuMonitorIO:679 : Triggering error callback</div><div>2012-08-02 11:44:07.564+0000: 4231: debug : qemuProcessHandleMonitorError:345 : Received error on 0x7fe7e4000e50 'milli'</div><div>2012-08-02 11:44:07.592+0000: 4232: debug : qemuDomainObjBeginJobInternal:753 : Starting job: async nested (async=migration out)</div><div>2012-08-02 11:44:07.593+0000: 4232: debug : qemuMonitorRef:239 : QEMU_MONITOR_REF: mon=0x7fe7e40c8410 refs=3</div><div>2012-08-02 11:44:07.593+0000: 4232: debug : qemuMonitorGetMigrationStatus:1776 : mon=0x7fe7e40c8410</div><div>2012-08-02 11:44:07.593+0000: 4232: debug : qemuMonitorSend:851 : Attempt to send command while error is set internal error Unexpected JSON reply '{"error": {"class": "JSONParsing", "desc": "Invalid JSON syntax", "data": {}}}'</div><div>2012-08-02 11:44:07.593+0000: 4232: debug : qemuMonitorUnref:248 : QEMU_MONITOR_UNREF: mon=0x7fe7e40c8410 refs=2</div><div>2012-08-02 11:44:07.593+0000: 4232: debug : doPeer2PeerMigrate3:2425 : Finish3 0x7fe7e00c4e50 ret=-1</div><div>2012-08-02 11:44:08.798+0000: 4233: debug : virDomainInterfaceStats:7299 : dom=0x7fe7e40c72d0, (VM: name=milli, uuid=2291f0d8-6341-4821-8774-2ed84b8bccc1), path=vnet0, stats=0x7fe803e45b10, size=64</div><div>2012-08-02 11:44:08.802+0000: 4233: debug : virDomainFree:2345 : dom=0x7fe7e40c72d0, (VM: name=milli, uuid=2291f0d8-6341-4821-8774-2ed84b8bccc1)</div><div>2012-08-02 11:44:12.564+0000: 4234: debug : virDomainGetJobInfo:16465 : dom=0x7fe7dc000e00, (VM: name=milli, uuid=2291f0d8-6341-4821-8774-2ed84b8bccc1), info=0x7fe803644af0</div><div>2012-08-02 11:44:12.564+0000: 4234: debug : virDomainFree:2345 : dom=0x7fe7dc000e00, (VM: name=milli, uuid=2291f0d8-6341-4821-8774-2ed84b8bccc1)</div><div>2012-08-02 11:44:13.804+0000: 4235: debug : virDomainGetInfo:4298 : dom=0x7fe7e8001bd0, (VM: name=milli, uuid=2291f0d8-6341-4821-8774-2ed84b8bccc1), info=0x7fe802e43b20</div><div>2012-08-02 11:44:13.805+0000: 4235: debug : qemudGetProcessInfo:1156 : Got status for 5448/0 user=1770 sys=1445 cpu=0 rss=185496</div><div>2012-08-02 11:44:13.805+0000: 4235: debug : virDomainFree:2345 : dom=0x7fe7e8001bd0, (VM: name=milli, uuid=2291f0d8-6341-4821-8774-2ed84b8bccc1)</div><div>2012-08-02 11:44:13.806+0000: 4238: debug : virDomainInterfaceStats:7299 : dom=0x7fe7f0002520, (VM: name=milli, uuid=2291f0d8-6341-4821-8774-2ed84b8bccc1), path=vnet0, stats=0x7fe801640b10, size=64</div><div>2012-08-02 11:44:13.807+0000: 4238: debug : virDomainFree:2345 : dom=0x7fe7f0002520, (VM: name=milli, uuid=2291f0d8-6341-4821-8774-2ed84b8bccc1)</div><div>2012-08-02 11:44:16.474+0000: 4232: error : virNetClientProgramDispatchError:174 : An error occurred, but the cause is unknown</div><div>2012-08-02 11:44:16.474+0000: 4232: debug : doPeer2PeerMigrate3:2458 : Confirm3 0x7fe7e40029c0 ret=-1 vm=0x7fe7e4000e50</div><div>2012-08-02 11:44:16.474+0000: 4232: debug : qemuMigrationConfirm:3109 : driver=0x7fe7f80bd9c0, conn=0x7fe7e40029c0, vm=0x7fe7e4000e50, cookiein=(null), cookieinlen=0, flags=3, retcode=1</div><div>2012-08-02 11:44:16.474+0000: 4232: debug : qemuMigrationEatCookie:752 : cookielen=0 cookie='(null)'</div><div>2012-08-02 11:44:16.475+0000: 4232: debug : qemuProcessStartCPUs:2644 : Using lock state '(null)'</div><div>2012-08-02 11:44:16.475+0000: 4232: debug : qemuDomainObjBeginJobInternal:753 : Starting job: async nested (async=migration out)</div></div><div><br></div><div>And this is from a migration gone well:</div><div><div>2012-08-02 11:41:28.386+0000: 618: debug : qemuProcessStop:3872 : Shutting down VM 'milli' pid=3776 flags=1</div><div>2012-08-02 11:41:28.386+0000: 618: debug : qemuMonitorClose:797 : QEMU_MONITOR_CLOSE: mon=0x7f49dc000d50 refs=2</div><div>2012-08-02 11:41:28.386+0000: 618: debug : qemuMonitorUnref:248 : QEMU_MONITOR_UNREF: mon=0x7f49dc000d50 refs=1</div><div>2012-08-02 11:41:28.386+0000: 606: debug : qemuMonitorUnref:248 : QEMU_MONITOR_UNREF: mon=0x7f49dc000d50 refs=0</div><div>2012-08-02 11:41:28.386+0000: 606: debug : qemuMonitorFree:225 : mon=0x7f49dc000d50</div><div>2012-08-02 11:41:28.386+0000: 618: debug : qemuProcessKill:3769 : vm=milli pid=3776 flags=5</div><div>2012-08-02 11:41:28.586+0000: 618: debug : qemuDomainCleanupRun:1921 : driver=0x7f49e806f1b0, vm=milli</div><div>2012-08-02 11:41:28.586+0000: 618: debug : qemuProcessAutoDestroyRemove:4320 : vm=milli</div><div>2012-08-02 11:41:28.586+0000: 618: debug : qemuDriverCloseCallbackUnset:578 : vm=milli, uuid=2291f0d8-6341-4821-8774-2ed84b8bccc1, cb=0x4900d0</div><div>2012-08-02 11:41:28.586+0000: 618: debug : virCgroupNew:603 : New group /libvirt/qemu/milli</div><div>2012-08-02 11:41:28.587+0000: 618: debug : virCgroupDetect:262 : Detected mount/mapping 0:cpu at /sys/fs/cgroup/cpu,cpuacct in /system/libvirtd.service</div><div>2012-08-02 11:41:28.587+0000: 618: debug : virCgroupDetect:262 : Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpu,cpuacct in /system/libvirtd.service</div><div>2012-08-02 11:41:28.587+0000: 618: debug : virCgroupDetect:262 : Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in</div><div>2012-08-02 11:41:28.587+0000: 618: debug : virCgroupDetect:262 : Detected mount/mapping 3:memory at /sys/fs/cgroup/memory in</div><div>2012-08-02 11:41:28.587+0000: 618: debug : virCgroupDetect:262 : Detected mount/mapping 4:devices at /sys/fs/cgroup/devices in</div><div>2012-08-02 11:41:28.587+0000: 618: debug : virCgroupDetect:262 : Detected mount/mapping 5:freezer at /sys/fs/cgroup/freezer in</div><div>2012-08-02 11:41:28.587+0000: 618: debug : virCgroupDetect:262 : Detected mount/mapping 6:blkio at /sys/fs/cgroup/blkio in</div><div>2012-08-02 11:41:28.587+0000: 618: debug : virCgroupMakeGroup:524 : Make group /libvirt/qemu/milli</div><div>2012-08-02 11:41:28.587+0000: 618: debug : virCgroupMakeGroup:546 : Make controller /sys/fs/cgroup/cpu,cpuacct/system/libvirtd.service/libvirt/qemu/milli/</div><div>2012-08-02 11:41:28.587+0000: 618: debug : virCgroupMakeGroup:546 : Make controller /sys/fs/cgroup/cpu,cpuacct/system/libvirtd.service/libvirt/qemu/milli/</div><div>2012-08-02 11:41:28.587+0000: 618: debug : virCgroupMakeGroup:546 : Make controller /sys/fs/cgroup/cpuset/libvirt/qemu/milli/</div><div>2012-08-02 11:41:28.587+0000: 618: debug : virCgroupMakeGroup:546 : Make controller /sys/fs/cgroup/memory/libvirt/qemu/milli/</div><div>2012-08-02 11:41:28.587+0000: 618: debug : virCgroupMakeGroup:546 : Make controller /sys/fs/cgroup/devices/libvirt/qemu/milli/</div><div>2012-08-02 11:41:28.587+0000: 618: debug : virCgroupMakeGroup:546 : Make controller /sys/fs/cgroup/freezer/libvirt/qemu/milli/</div><div>2012-08-02 11:41:28.587+0000: 618: debug : virCgroupMakeGroup:546 : Make controller /sys/fs/cgroup/blkio/libvirt/qemu/milli/</div><div>2012-08-02 11:41:28.587+0000: 618: debug : virCgroupRemove:758 : Removing cgroup /sys/fs/cgroup/cpu,cpuacct/system/libvirtd.service/libvirt/qemu/milli/ and all child cgroups</div><div>2012-08-02 11:41:28.588+0000: 618: debug : virCgroupRemoveRecursively:713 : Removing cgroup /sys/fs/cgroup/cpu,cpuacct/system/libvirtd.service/libvirt/qemu/milli//vcpu1</div><div>2012-08-02 11:41:28.595+0000: 618: debug : virCgroupRemoveRecursively:713 : Removing cgroup /sys/fs/cgroup/cpu,cpuacct/system/libvirtd.service/libvirt/qemu/milli//vcpu0</div><div>2012-08-02 11:41:28.603+0000: 618: debug : virCgroupRemoveRecursively:713 : Removing cgroup /sys/fs/cgroup/cpu,cpuacct/system/libvirtd.service/libvirt/qemu/milli/</div><div>2012-08-02 11:41:28.611+0000: 618: debug : virCgroupRemove:758 : Removing cgroup /sys/fs/cgroup/cpu,cpuacct/system/libvirtd.service/libvirt/qemu/milli/ and all child cgroups</div><div>2012-08-02 11:41:28.611+0000: 618: debug : virCgroupRemove:758 : Removing cgroup /sys/fs/cgroup/cpuset/libvirt/qemu/milli/ and all child cgroups</div><div>2012-08-02 11:41:28.611+0000: 618: debug : virCgroupRemoveRecursively:713 : Removing cgroup /sys/fs/cgroup/cpuset/libvirt/qemu/milli/</div><div>2012-08-02 11:41:28.620+0000: 618: debug : virCgroupRemove:758 : Removing cgroup /sys/fs/cgroup/memory/libvirt/qemu/milli/ and all child cgroups</div><div>2012-08-02 11:41:28.620+0000: 618: debug : virCgroupRemoveRecursively:713 : Removing cgroup /sys/fs/cgroup/memory/libvirt/qemu/milli/</div><div>2012-08-02 11:41:28.633+0000: 618: debug : virCgroupRemove:758 : Removing cgroup /sys/fs/cgroup/devices/libvirt/qemu/milli/ and all child cgroups</div><div>2012-08-02 11:41:28.679+0000: 618: debug : virCgroupRemoveRecursively:713 : Removing cgroup /sys/fs/cgroup/devices/libvirt/qemu/milli/</div><div>2012-08-02 11:41:28.684+0000: 618: debug : virCgroupRemove:758 : Removing cgroup /sys/fs/cgroup/freezer/libvirt/qemu/milli/ and all child cgroups</div><div>2012-08-02 11:41:28.684+0000: 618: debug : virCgroupRemoveRecursively:713 : Removing cgroup /sys/fs/cgroup/freezer/libvirt/qemu/milli/</div><div>2012-08-02 11:41:28.689+0000: 618: debug : virCgroupRemove:758 : Removing cgroup /sys/fs/cgroup/blkio/libvirt/qemu/milli/ and all child cgroups</div><div>2012-08-02 11:41:28.689+0000: 618: debug : virCgroupRemoveRecursively:713 : Removing cgroup /sys/fs/cgroup/blkio/libvirt/qemu/milli/</div><div>2012-08-02 11:41:28.697+0000: 618: debug : virConnectClose:1496 : conn=0x7f49dc10fac0</div><div>2012-08-02 11:41:28.698+0000: 606: debug : virDomainFree:2345 : dom=0xaa0ea0, (VM: name=milli, uuid=2291f0d8-6341-4821-8774-2ed84b8bccc1)</div><div>2012-08-02 11:41:28.700+0000: 618: debug : qemuDomainObjEndAsyncJob:887 : Stopping async job: migration out</div></div><div><br></div><div><div><div># rpm -qa | egrep '(vdsm|libvirt|sanlock|json)' | sort -d</div><div>json-c-0.9-4.fc17.x86_64</div><div>json-glib-0.14.2-2.fc17.x86_64</div><div>libvirt-0.9.11.4-3.fc17.x86_64</div><div>libvirt-client-0.9.11.4-3.fc17.x86_64</div><div>libvirt-daemon-0.9.11.4-3.fc17.x86_64</div><div>libvirt-daemon-config-network-0.9.11.4-3.fc17.x86_64</div><div>libvirt-daemon-config-nwfilter-0.9.11.4-3.fc17.x86_64</div><div>libvirt-lock-sanlock-0.9.11.4-3.fc17.x86_64</div><div>libvirt-python-0.9.11.4-3.fc17.x86_64</div><div>python-simplejson-2.5.2-1.fc17.x86_64</div><div>sanlock-2.3-3.fc17.x86_64</div><div>sanlock-lib-2.3-3.fc17.x86_64</div><div>sanlock-python-2.3-3.fc17.x86_64</div><div>vdsm-4.10.0-5.fc17.x86_64</div><div>vdsm-cli-4.10.0-5.fc17.noarch</div><div>vdsm-python-4.10.0-5.fc17.x86_64</div><div>vdsm-xmlrpc-4.10.0-5.fc17.noarch</div></div></div><div>
<div><br class="Apple-interchange-newline"><br></div><div>Best Regards<br>-------------------------------------------------------------------------------<br>Karli Sjöberg<br>Swedish University of Agricultural Sciences<br>Box 7079 (Visiting Address Kronåsvägen 8)<br>S-750 07 Uppsala, Sweden<br>Phone: +46-(0)18-67 15 66</div><div><a href="mailto:karli.sjoberg@adm.slu.se">karli.sjoberg@slu.se</a></div>
</div>
<br></div></body></html>