check selinux On 08/02/2012 03:01 PM, Karli Sjöberg wrote:
Hi,
Wondering if anyone has encountered the same issue as me. On one host in my cluster, if I migrate in a guest, I cannot migrate it out to another host? The get "stuck" there, so to speak. Same when a guest is started on that particular host, it is impossible to migrate them out again.
iptables are flushed, for excluding purposes. vdsmd, libvirtd and sanlock are all running.
This is what I caught from libvirtd.log: 2012-08-02 11:44:07.542+0000: 4231: debug : qemuMonitorUnref:248 : QEMU_MONITOR_UNREF: mon=0x7fe7e40c8410 refs=3 2012-08-02 11:44:07.542+0000: 4232: debug : qemuMonitorUnref:248 : QEMU_MONITOR_UNREF: mon=0x7fe7e40c8410 refs=2 2012-08-02 11:44:07.542+0000: 4232: debug : virConnectIsAlive:18395 : conn=0x7fe7e00c4e50 2012-08-02 11:44:07.562+0000: 4236: debug : virDomainMigrateSetMaxDowntime:16565 : dom=0x7fe7d8000cc0, (VM: name=milli, uuid=2291f0d8-6341-4821-8774-2ed84b8bccc1), downtime=50, flags=0 2012-08-02 11:44:07.562+0000: 4236: debug : qemuDomainObjBeginJobInternal:753 : Starting job: migration operation (async=migration out) 2012-08-02 11:44:07.562+0000: 4236: debug : qemuDomainMigrateSetMaxDowntime:9500 : Setting migration downtime to 50ms 2012-08-02 11:44:07.562+0000: 4236: debug : qemuMonitorRef:239 : QEMU_MONITOR_REF: mon=0x7fe7e40c8410 refs=3 2012-08-02 11:44:07.562+0000: 4236: debug : qemuMonitorSetMigrationDowntime:1753 : mon=0x7fe7e40c8410 downtime=50 2012-08-02 11:44:07.562+0000: 4236: debug : qemuMonitorSend:861 : QEMU_MONITOR_SEND_MSG: mon=0x7fe7e40c8410 msg={"execute":"migrate_set_downtime","arguments":{"value":0,050000},"id":"libvirt-261"} fd=-1 2012-08-02 11:44:07.563+0000: 4231: debug : qemuMonitorRef:239 : QEMU_MONITOR_REF: mon=0x7fe7e40c8410 refs=4 2012-08-02 11:44:07.563+0000: 4231: debug : qemuMonitorIOWrite:470 : QEMU_MONITOR_IO_WRITE: mon=0x7fe7e40c8410 buf={"execute":"migrate_set_downtime","arguments":{"value":0,050000},"id":"libvirt-261"} len=86 ret=86 errno=11 2012-08-02 11:44:07.563+0000: 4231: debug : qemuMonitorUnref:248 : QEMU_MONITOR_UNREF: mon=0x7fe7e40c8410 refs=3 2012-08-02 11:44:07.564+0000: 4231: debug : qemuMonitorRef:239 : QEMU_MONITOR_REF: mon=0x7fe7e40c8410 refs=4 2012-08-02 11:44:07.564+0000: 4231: debug : qemuMonitorIOProcess:365 : QEMU_MONITOR_IO_PROCESS: mon=0x7fe7e40c8410 buf={"error": {"class": "JSONParsing", "desc": "Invalid JSON syntax", "data": {}}} len=80 2012-08-02 11:44:07.564+0000: 4231: debug : qemuMonitorUnref:248 : QEMU_MONITOR_UNREF: mon=0x7fe7e40c8410 refs=3 2012-08-02 11:44:07.564+0000: 4236: error : qemuMonitorJSONCheckError:331 : internal error unable to execute QEMU command 'migrate_set_downtime': Invalid JSON syntax 2012-08-02 11:44:07.564+0000: 4236: debug : qemuMonitorUnref:248 : QEMU_MONITOR_UNREF: mon=0x7fe7e40c8410 refs=2 2012-08-02 11:44:07.564+0000: 4236: debug : qemuDomainObjEndJob:870 : Stopping job: migration operation (async=migration out) 2012-08-02 11:44:07.564+0000: 4231: debug : qemuMonitorRef:239 : QEMU_MONITOR_REF: mon=0x7fe7e40c8410 refs=3 2012-08-02 11:44:07.564+0000: 4231: debug : qemuMonitorIOProcess:365 : QEMU_MONITOR_IO_PROCESS: mon=0x7fe7e40c8410 buf={"error": {"class": "JSONParsing", "desc": "Invalid JSON syntax", "data": {}}} len=80 2012-08-02 11:44:07.564+0000: 4236: debug : virDomainFree:2345 : dom=0x7fe7d8000cc0, (VM: name=milli, uuid=2291f0d8-6341-4821-8774-2ed84b8bccc1) 2012-08-02 11:44:07.564+0000: 4231: error : qemuMonitorJSONIOProcessLine:156 : internal error Unexpected JSON reply '{"error": {"class": "JSONParsing", "desc": "Invalid JSON syntax", "data": {}}}' 2012-08-02 11:44:07.564+0000: 4231: debug : qemuMonitorIO:645 : Error on monitor internal error Unexpected JSON reply '{"error": {"class": "JSONParsing", "desc": "Invalid JSON syntax", "data": {}}}' 2012-08-02 11:44:07.564+0000: 4231: debug : qemuMonitorUnref:248 : QEMU_MONITOR_UNREF: mon=0x7fe7e40c8410 refs=2 2012-08-02 11:44:07.564+0000: 4231: debug : qemuMonitorIO:679 : Triggering error callback 2012-08-02 11:44:07.564+0000: 4231: debug : qemuProcessHandleMonitorError:345 : Received error on 0x7fe7e4000e50 'milli' 2012-08-02 11:44:07.592+0000: 4232: debug : qemuDomainObjBeginJobInternal:753 : Starting job: async nested (async=migration out) 2012-08-02 11:44:07.593+0000: 4232: debug : qemuMonitorRef:239 : QEMU_MONITOR_REF: mon=0x7fe7e40c8410 refs=3 2012-08-02 11:44:07.593+0000: 4232: debug : qemuMonitorGetMigrationStatus:1776 : mon=0x7fe7e40c8410 2012-08-02 11:44:07.593+0000: 4232: debug : qemuMonitorSend:851 : Attempt to send command while error is set internal error Unexpected JSON reply '{"error": {"class": "JSONParsing", "desc": "Invalid JSON syntax", "data": {}}}' 2012-08-02 11:44:07.593+0000: 4232: debug : qemuMonitorUnref:248 : QEMU_MONITOR_UNREF: mon=0x7fe7e40c8410 refs=2 2012-08-02 11:44:07.593+0000: 4232: debug : doPeer2PeerMigrate3:2425 : Finish3 0x7fe7e00c4e50 ret=-1 2012-08-02 11:44:08.798+0000: 4233: debug : virDomainInterfaceStats:7299 : dom=0x7fe7e40c72d0, (VM: name=milli, uuid=2291f0d8-6341-4821-8774-2ed84b8bccc1), path=vnet0, stats=0x7fe803e45b10, size=64 2012-08-02 11:44:08.802+0000: 4233: debug : virDomainFree:2345 : dom=0x7fe7e40c72d0, (VM: name=milli, uuid=2291f0d8-6341-4821-8774-2ed84b8bccc1) 2012-08-02 11:44:12.564+0000: 4234: debug : virDomainGetJobInfo:16465 : dom=0x7fe7dc000e00, (VM: name=milli, uuid=2291f0d8-6341-4821-8774-2ed84b8bccc1), info=0x7fe803644af0 2012-08-02 11:44:12.564+0000: 4234: debug : virDomainFree:2345 : dom=0x7fe7dc000e00, (VM: name=milli, uuid=2291f0d8-6341-4821-8774-2ed84b8bccc1) 2012-08-02 11:44:13.804+0000: 4235: debug : virDomainGetInfo:4298 : dom=0x7fe7e8001bd0, (VM: name=milli, uuid=2291f0d8-6341-4821-8774-2ed84b8bccc1), info=0x7fe802e43b20 2012-08-02 11:44:13.805+0000: 4235: debug : qemudGetProcessInfo:1156 : Got status for 5448/0 user=1770 sys=1445 cpu=0 rss=185496 2012-08-02 11:44:13.805+0000: 4235: debug : virDomainFree:2345 : dom=0x7fe7e8001bd0, (VM: name=milli, uuid=2291f0d8-6341-4821-8774-2ed84b8bccc1) 2012-08-02 11:44:13.806+0000: 4238: debug : virDomainInterfaceStats:7299 : dom=0x7fe7f0002520, (VM: name=milli, uuid=2291f0d8-6341-4821-8774-2ed84b8bccc1), path=vnet0, stats=0x7fe801640b10, size=64 2012-08-02 11:44:13.807+0000: 4238: debug : virDomainFree:2345 : dom=0x7fe7f0002520, (VM: name=milli, uuid=2291f0d8-6341-4821-8774-2ed84b8bccc1) 2012-08-02 11:44:16.474+0000: 4232: error : virNetClientProgramDispatchError:174 : An error occurred, but the cause is unknown 2012-08-02 11:44:16.474+0000: 4232: debug : doPeer2PeerMigrate3:2458 : Confirm3 0x7fe7e40029c0 ret=-1 vm=0x7fe7e4000e50 2012-08-02 11:44:16.474+0000: 4232: debug : qemuMigrationConfirm:3109 : driver=0x7fe7f80bd9c0, conn=0x7fe7e40029c0, vm=0x7fe7e4000e50, cookiein=(null), cookieinlen=0, flags=3, retcode=1 2012-08-02 11:44:16.474+0000: 4232: debug : qemuMigrationEatCookie:752 : cookielen=0 cookie='(null)' 2012-08-02 11:44:16.475+0000: 4232: debug : qemuProcessStartCPUs:2644 : Using lock state '(null)' 2012-08-02 11:44:16.475+0000: 4232: debug : qemuDomainObjBeginJobInternal:753 : Starting job: async nested (async=migration out)
And this is from a migration gone well: 2012-08-02 11:41:28.386+0000: 618: debug : qemuProcessStop:3872 : Shutting down VM 'milli' pid=3776 flags=1 2012-08-02 11:41:28.386+0000: 618: debug : qemuMonitorClose:797 : QEMU_MONITOR_CLOSE: mon=0x7f49dc000d50 refs=2 2012-08-02 11:41:28.386+0000: 618: debug : qemuMonitorUnref:248 : QEMU_MONITOR_UNREF: mon=0x7f49dc000d50 refs=1 2012-08-02 11:41:28.386+0000: 606: debug : qemuMonitorUnref:248 : QEMU_MONITOR_UNREF: mon=0x7f49dc000d50 refs=0 2012-08-02 11:41:28.386+0000: 606: debug : qemuMonitorFree:225 : mon=0x7f49dc000d50 2012-08-02 11:41:28.386+0000: 618: debug : qemuProcessKill:3769 : vm=milli pid=3776 flags=5 2012-08-02 11:41:28.586+0000: 618: debug : qemuDomainCleanupRun:1921 : driver=0x7f49e806f1b0, vm=milli 2012-08-02 11:41:28.586+0000: 618: debug : qemuProcessAutoDestroyRemove:4320 : vm=milli 2012-08-02 11:41:28.586+0000: 618: debug : qemuDriverCloseCallbackUnset:578 : vm=milli, uuid=2291f0d8-6341-4821-8774-2ed84b8bccc1, cb=0x4900d0 2012-08-02 11:41:28.586+0000: 618: debug : virCgroupNew:603 : New group /libvirt/qemu/milli 2012-08-02 11:41:28.587+0000: 618: debug : virCgroupDetect:262 : Detected mount/mapping 0:cpu at /sys/fs/cgroup/cpu,cpuacct in /system/libvirtd.service 2012-08-02 11:41:28.587+0000: 618: debug : virCgroupDetect:262 : Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpu,cpuacct in /system/libvirtd.service 2012-08-02 11:41:28.587+0000: 618: debug : virCgroupDetect:262 : Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in 2012-08-02 11:41:28.587+0000: 618: debug : virCgroupDetect:262 : Detected mount/mapping 3:memory at /sys/fs/cgroup/memory in 2012-08-02 11:41:28.587+0000: 618: debug : virCgroupDetect:262 : Detected mount/mapping 4:devices at /sys/fs/cgroup/devices in 2012-08-02 11:41:28.587+0000: 618: debug : virCgroupDetect:262 : Detected mount/mapping 5:freezer at /sys/fs/cgroup/freezer in 2012-08-02 11:41:28.587+0000: 618: debug : virCgroupDetect:262 : Detected mount/mapping 6:blkio at /sys/fs/cgroup/blkio in 2012-08-02 11:41:28.587+0000: 618: debug : virCgroupMakeGroup:524 : Make group /libvirt/qemu/milli 2012-08-02 11:41:28.587+0000: 618: debug : virCgroupMakeGroup:546 : Make controller /sys/fs/cgroup/cpu,cpuacct/system/libvirtd.service/libvirt/qemu/milli/ 2012-08-02 11:41:28.587+0000: 618: debug : virCgroupMakeGroup:546 : Make controller /sys/fs/cgroup/cpu,cpuacct/system/libvirtd.service/libvirt/qemu/milli/ 2012-08-02 11:41:28.587+0000: 618: debug : virCgroupMakeGroup:546 : Make controller /sys/fs/cgroup/cpuset/libvirt/qemu/milli/ 2012-08-02 11:41:28.587+0000: 618: debug : virCgroupMakeGroup:546 : Make controller /sys/fs/cgroup/memory/libvirt/qemu/milli/ 2012-08-02 11:41:28.587+0000: 618: debug : virCgroupMakeGroup:546 : Make controller /sys/fs/cgroup/devices/libvirt/qemu/milli/ 2012-08-02 11:41:28.587+0000: 618: debug : virCgroupMakeGroup:546 : Make controller /sys/fs/cgroup/freezer/libvirt/qemu/milli/ 2012-08-02 11:41:28.587+0000: 618: debug : virCgroupMakeGroup:546 : Make controller /sys/fs/cgroup/blkio/libvirt/qemu/milli/ 2012-08-02 11:41:28.587+0000: 618: debug : virCgroupRemove:758 : Removing cgroup /sys/fs/cgroup/cpu,cpuacct/system/libvirtd.service/libvirt/qemu/milli/ and all child cgroups 2012-08-02 11:41:28.588+0000: 618: debug : virCgroupRemoveRecursively:713 : Removing cgroup /sys/fs/cgroup/cpu,cpuacct/system/libvirtd.service/libvirt/qemu/milli//vcpu1 2012-08-02 11:41:28.595+0000: 618: debug : virCgroupRemoveRecursively:713 : Removing cgroup /sys/fs/cgroup/cpu,cpuacct/system/libvirtd.service/libvirt/qemu/milli//vcpu0 2012-08-02 11:41:28.603+0000: 618: debug : virCgroupRemoveRecursively:713 : Removing cgroup /sys/fs/cgroup/cpu,cpuacct/system/libvirtd.service/libvirt/qemu/milli/ 2012-08-02 11:41:28.611+0000: 618: debug : virCgroupRemove:758 : Removing cgroup /sys/fs/cgroup/cpu,cpuacct/system/libvirtd.service/libvirt/qemu/milli/ and all child cgroups 2012-08-02 11:41:28.611+0000: 618: debug : virCgroupRemove:758 : Removing cgroup /sys/fs/cgroup/cpuset/libvirt/qemu/milli/ and all child cgroups 2012-08-02 11:41:28.611+0000: 618: debug : virCgroupRemoveRecursively:713 : Removing cgroup /sys/fs/cgroup/cpuset/libvirt/qemu/milli/ 2012-08-02 11:41:28.620+0000: 618: debug : virCgroupRemove:758 : Removing cgroup /sys/fs/cgroup/memory/libvirt/qemu/milli/ and all child cgroups 2012-08-02 11:41:28.620+0000: 618: debug : virCgroupRemoveRecursively:713 : Removing cgroup /sys/fs/cgroup/memory/libvirt/qemu/milli/ 2012-08-02 11:41:28.633+0000: 618: debug : virCgroupRemove:758 : Removing cgroup /sys/fs/cgroup/devices/libvirt/qemu/milli/ and all child cgroups 2012-08-02 11:41:28.679+0000: 618: debug : virCgroupRemoveRecursively:713 : Removing cgroup /sys/fs/cgroup/devices/libvirt/qemu/milli/ 2012-08-02 11:41:28.684+0000: 618: debug : virCgroupRemove:758 : Removing cgroup /sys/fs/cgroup/freezer/libvirt/qemu/milli/ and all child cgroups 2012-08-02 11:41:28.684+0000: 618: debug : virCgroupRemoveRecursively:713 : Removing cgroup /sys/fs/cgroup/freezer/libvirt/qemu/milli/ 2012-08-02 11:41:28.689+0000: 618: debug : virCgroupRemove:758 : Removing cgroup /sys/fs/cgroup/blkio/libvirt/qemu/milli/ and all child cgroups 2012-08-02 11:41:28.689+0000: 618: debug : virCgroupRemoveRecursively:713 : Removing cgroup /sys/fs/cgroup/blkio/libvirt/qemu/milli/ 2012-08-02 11:41:28.697+0000: 618: debug : virConnectClose:1496 : conn=0x7f49dc10fac0 2012-08-02 11:41:28.698+0000: 606: debug : virDomainFree:2345 : dom=0xaa0ea0, (VM: name=milli, uuid=2291f0d8-6341-4821-8774-2ed84b8bccc1) 2012-08-02 11:41:28.700+0000: 618: debug : qemuDomainObjEndAsyncJob:887 : Stopping async job: migration out
# rpm -qa | egrep '(vdsm|libvirt|sanlock|json)' | sort -d json-c-0.9-4.fc17.x86_64 json-glib-0.14.2-2.fc17.x86_64 libvirt-0.9.11.4-3.fc17.x86_64 libvirt-client-0.9.11.4-3.fc17.x86_64 libvirt-daemon-0.9.11.4-3.fc17.x86_64 libvirt-daemon-config-network-0.9.11.4-3.fc17.x86_64 libvirt-daemon-config-nwfilter-0.9.11.4-3.fc17.x86_64 libvirt-lock-sanlock-0.9.11.4-3.fc17.x86_64 libvirt-python-0.9.11.4-3.fc17.x86_64 python-simplejson-2.5.2-1.fc17.x86_64 sanlock-2.3-3.fc17.x86_64 sanlock-lib-2.3-3.fc17.x86_64 sanlock-python-2.3-3.fc17.x86_64 vdsm-4.10.0-5.fc17.x86_64 vdsm-cli-4.10.0-5.fc17.noarch vdsm-python-4.10.0-5.fc17.x86_64 vdsm-xmlrpc-4.10.0-5.fc17.noarch
Best Regards ------------------------------------------------------------------------------- Karli Sjöberg Swedish University of Agricultural Sciences Box 7079 (Visiting Address Kronåsvägen 8) S-750 07 Uppsala, Sweden Phone: +46-(0)18-67 15 66 karli.sjoberg@slu.se <mailto:karli.sjoberg@adm.slu.se>
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Dafna Ron