2017-08-25 14:21:28,159+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000289728', 'lastCheck': '0.2', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000644597', 'lastCheck': '2.7', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000291946', 'lastCheck': '2.7', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000706921', 'lastCheck': '3.2', 'valid': True}} from=internal, task_id=8cdd5a4c-51fb-45b8-89c4-fd1ffb8ff74e (api:52) 2017-08-25 14:21:29,711+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:21:40,992+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:21:40,994+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmIoTunePolicies succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:21:42,616+0200 INFO (jsonrpc/1) [vdsm.api] START repoStats(options=None) from=::ffff:172.16.252.200,56136, flow_id=3dca77f2, task_id=5cf7704d-ff96-458b-a52a-04d66b26bce4 (api:46) 2017-08-25 14:21:42,616+0200 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000314141', 'lastCheck': '4.6', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000819328', 'lastCheck': '7.2', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000351961', 'lastCheck': '7.1', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000782557', 'lastCheck': '7.6', 'valid': True}} from=::ffff:172.16.252.200,56136, flow_id=3dca77f2, task_id=5cf7704d-ff96-458b-a52a-04d66b26bce4 (api:52) 2017-08-25 14:21:42,622+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:539) 2017-08-25 14:21:43,237+0200 INFO (periodic/2) [vdsm.api] START repoStats(options=None) from=internal, task_id=7a2809ec-51aa-43c9-9131-acbd34176682 (api:46) 2017-08-25 14:21:43,237+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000305612', 'lastCheck': '0.2', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000604943', 'lastCheck': '0.2', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00028699', 'lastCheck': '0.3', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000759608', 'lastCheck': '0.2', 'valid': True}} from=internal, task_id=7a2809ec-51aa-43c9-9131-acbd34176682 (api:52) 2017-08-25 14:21:44,996+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:21:56,000+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:539) 2017-08-25 14:21:56,011+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmIoTunePolicies succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:21:57,772+0200 INFO (jsonrpc/0) [vdsm.api] START repoStats(options=None) from=::ffff:172.16.252.200,56136, flow_id=18cbb200, task_id=2bf1b372-7045-48a0-a708-5b67a997e50f (api:46) 2017-08-25 14:21:57,772+0200 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00032868', 'lastCheck': '4.7', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.0007566', 'lastCheck': '2.3', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000334003', 'lastCheck': '2.3', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000736448', 'lastCheck': '2.8', 'valid': True}} from=::ffff:172.16.252.200,56136, flow_id=18cbb200, task_id=2bf1b372-7045-48a0-a708-5b67a997e50f (api:52) 2017-08-25 14:21:57,778+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:21:57,931+0200 INFO (jsonrpc/4) [vdsm.api] START migrate(params={u'incomingLimit': 2, u'src': u'microcloud22.rxmgmt.databay.de', u'dstqemu': u'172.16.252.124', u'autoConverge': u'true', u'tunneled': u'false', u'enableGuestEvents': True, u'dst': u'microcloud24.rxmgmt.databay.de:54321', u'convergenceSchedule': {u'init': [{u'params': [u'100'], u'name': u'setDowntime'}], u'stalling': [{u'action': {u'params': [u'150'], u'name': u'setDowntime'}, u'limit': 1}, {u'action': {u'params': [u'200'], u'name': u'setDowntime'}, u'limit': 2}, {u'action': {u'params': [u'300'], u'name': u'setDowntime'}, u'limit': 3}, {u'action': {u'params': [u'400'], u'name': u'setDowntime'}, u'limit': 4}, {u'action': {u'params': [u'500'], u'name': u'setDowntime'}, u'limit': 6}, {u'action': {u'params': [], u'name': u'abort'}, u'limit': -1}]}, u'vmId': u'ffb71f79-54cd-4f0e-b6b5-3670236cb497', u'abortOnError': u'true', u'outgoingLimit': 2, u'compressed': u'false', u'method': u'online', 'mode': 'remote'}) from=::ffff:172.16.252.200,56136, flow_id=a1cd6200-33c0-43f5-b1ff-a7a38b6b6f8a (api:46) 2017-08-25 14:21:57,932+0200 INFO (jsonrpc/4) [vdsm.api] FINISH migrate return={'status': {'message': 'Migration in progress', 'code': 0}, 'progress': 0} from=::ffff:172.16.252.200,56136, flow_id=a1cd6200-33c0-43f5-b1ff-a7a38b6b6f8a (api:52) 2017-08-25 14:21:57,932+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.migrate succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:21:58,315+0200 INFO (periodic/2) [vdsm.api] START repoStats(options=None) from=internal, task_id=c10d0c59-9e14-46f0-9829-2cc7ab3a0b0f (api:46) 2017-08-25 14:21:58,316+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00032868', 'lastCheck': '0.3', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.0007566', 'lastCheck': '2.8', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000334003', 'lastCheck': '2.8', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000736448', 'lastCheck': '3.3', 'valid': True}} from=internal, task_id=c10d0c59-9e14-46f0-9829-2cc7ab3a0b0f (api:52) 2017-08-25 14:21:58,363+0200 INFO (migsrc/ffb71f79) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') Creation of destination VM took: 0 seconds (migration:455) 2017-08-25 14:21:58,363+0200 INFO (migsrc/ffb71f79) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') starting migration to qemu+tls://microcloud24.rxmgmt.databay.de/system with miguri tcp://172.16.252.124 (migration:480) 2017-08-25 14:21:58,538+0200 ERROR (migsrc/ffb71f79) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') failed to initialize gluster connection (src=0x7fd81c06afa0 priv=0x7fd81c0142d0): Success (migration:287) 2017-08-25 14:21:58,574+0200 ERROR (migsrc/ffb71f79) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') Failed to migrate (migration:429) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 411, in run self._startUnderlyingMigration(time.time()) File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 487, in _startUnderlyingMigration self._perform_with_conv_schedule(duri, muri) File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 563, in _perform_with_conv_schedule self._perform_migration(duri, muri) File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 529, in _perform_migration self._vm._dom.migrateToURI3(duri, params, flags) File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 69, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 123, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 944, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1939, in migrateToURI3 if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) libvirtError: failed to initialize gluster connection (src=0x7fd81c06afa0 priv=0x7fd81c0142d0): Success 2017-08-25 14:22:00,208+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:22:01,386+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call VM.getMigrationStatus succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:22:11,004+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:22:11,012+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmIoTunePolicies succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:22:11,949+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID=u'5d99af76-33b5-47d8-99da-1f32413c7bb0', spUUID=u'00000001-0001-0001-0001-0000000000b9', imgUUID=u'9c007b27-0ab7-4474-9317-a294fd04c65f', volUUID=u'ba2dfe7f-bdcf-4680-878d-7d71272f3ffb', options=None) from=internal, task_id=8d550498-24b9-4e1d-95e3-28db51a2de23 (api:46) 2017-08-25 14:22:11,952+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '8133296128', 'apparentsize': '10737418240'} from=internal, task_id=8d550498-24b9-4e1d-95e3-28db51a2de23 (api:52) 2017-08-25 14:22:13,064+0200 INFO (jsonrpc/6) [vdsm.api] START repoStats(options=None) from=::ffff:172.16.252.200,56136, flow_id=6bc62149, task_id=1bcca07e-32f0-40ed-a188-680bfcf1f733 (api:46) 2017-08-25 14:22:13,064+0200 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00034783', 'lastCheck': '0.1', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000652776', 'lastCheck': '0.1', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000320179', 'lastCheck': '0.1', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000688439', 'lastCheck': '0.0', 'valid': True}} from=::ffff:172.16.252.200,56136, flow_id=6bc62149, task_id=1bcca07e-32f0-40ed-a188-680bfcf1f733 (api:52) 2017-08-25 14:22:13,068+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:22:13,393+0200 INFO (periodic/2) [vdsm.api] START repoStats(options=None) from=internal, task_id=f6a5c487-2d6b-4536-a00a-bdd33c1f0936 (api:46) 2017-08-25 14:22:13,393+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00034783', 'lastCheck': '0.4', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000652776', 'lastCheck': '0.4', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000320179', 'lastCheck': '0.4', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000688439', 'lastCheck': '0.4', 'valid': True}} from=internal, task_id=f6a5c487-2d6b-4536-a00a-bdd33c1f0936 (api:52) 2017-08-25 14:22:16,364+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:22:18,029+0200 INFO (itmap/0) [IOProcessClient] Starting client ioprocess-1194 (__init__:330) 2017-08-25 14:22:18,035+0200 INFO (itmap/1) [IOProcessClient] Starting client ioprocess-1195 (__init__:330) 2017-08-25 14:22:18,040+0200 INFO (ioprocess/15193) [IOProcess] Starting ioprocess (__init__:452) 2017-08-25 14:22:18,042+0200 INFO (ioprocess/15199) [IOProcess] Starting ioprocess (__init__:452) 2017-08-25 14:22:18,050+0200 INFO (monitor/7c8deaa) [storage.StorageDomain] Resource namespace 01_img_7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96 already registered (sd:727) 2017-08-25 14:22:18,050+0200 INFO (monitor/7c8deaa) [storage.StorageDomain] Resource namespace 02_vol_7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96 already registered (sd:736) 2017-08-25 14:22:26,008+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:22:26,014+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmIoTunePolicies succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:22:28,447+0200 INFO (periodic/2) [vdsm.api] START repoStats(options=None) from=internal, task_id=0ca5df30-45fa-47e1-b0f3-1413f49cc404 (api:46) 2017-08-25 14:22:28,447+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000300638', 'lastCheck': '0.4', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000787794', 'lastCheck': '3.0', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000290231', 'lastCheck': '2.9', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000722204', 'lastCheck': '3.5', 'valid': True}} from=internal, task_id=0ca5df30-45fa-47e1-b0f3-1413f49cc404 (api:52) 2017-08-25 14:22:29,243+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(options=None) from=::ffff:172.16.252.200,56136, flow_id=47a961c7, task_id=9d316b34-913e-4bf4-a1a7-2e1cc0487399 (api:46) 2017-08-25 14:22:29,243+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000300638', 'lastCheck': '1.2', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000787794', 'lastCheck': '3.8', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000290231', 'lastCheck': '3.7', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000722204', 'lastCheck': '4.2', 'valid': True}} from=::ffff:172.16.252.200,56136, flow_id=47a961c7, task_id=9d316b34-913e-4bf4-a1a7-2e1cc0487399 (api:52) 2017-08-25 14:22:29,247+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:22:31,813+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:22:35,527+0200 INFO (monitor/5d99af7) [storage.StorageDomain] Resource namespace 01_img_5d99af76-33b5-47d8-99da-1f32413c7bb0 already registered (sd:727) 2017-08-25 14:22:35,528+0200 INFO (monitor/5d99af7) [storage.StorageDomain] Resource namespace 02_vol_5d99af76-33b5-47d8-99da-1f32413c7bb0 already registered (sd:736) 2017-08-25 14:22:38,818+0200 INFO (jsonrpc/1) [vdsm.api] START migrate(params={u'incomingLimit': 2, u'src': u'microcloud22.rxmgmt.databay.de', u'dstqemu': u'172.16.252.128', u'autoConverge': u'true', u'tunneled': u'false', u'enableGuestEvents': True, u'dst': u'microcloud28.rxmgmt.databay.de:54321', u'convergenceSchedule': {u'init': [{u'params': [u'100'], u'name': u'setDowntime'}], u'stalling': [{u'action': {u'params': [u'150'], u'name': u'setDowntime'}, u'limit': 1}, {u'action': {u'params': [u'200'], u'name': u'setDowntime'}, u'limit': 2}, {u'action': {u'params': [u'300'], u'name': u'setDowntime'}, u'limit': 3}, {u'action': {u'params': [u'400'], u'name': u'setDowntime'}, u'limit': 4}, {u'action': {u'params': [u'500'], u'name': u'setDowntime'}, u'limit': 6}, {u'action': {u'params': [], u'name': u'abort'}, u'limit': -1}]}, u'vmId': u'ffb71f79-54cd-4f0e-b6b5-3670236cb497', u'abortOnError': u'true', u'outgoingLimit': 2, u'compressed': u'false', u'method': u'online', 'mode': 'remote'}) from=::ffff:172.16.252.200,56136, flow_id=c40bedb6-dff9-4204-8c7a-cf87247ac90e (api:46) 2017-08-25 14:22:38,820+0200 INFO (jsonrpc/1) [vdsm.api] FINISH migrate return={'status': {'message': 'Migration in progress', 'code': 0}, 'progress': 0} from=::ffff:172.16.252.200,56136, flow_id=c40bedb6-dff9-4204-8c7a-cf87247ac90e (api:52) 2017-08-25 14:22:38,820+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call VM.migrate succeeded in 0.01 seconds (__init__:539) 2017-08-25 14:22:39,347+0200 INFO (migsrc/ffb71f79) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') Creation of destination VM took: 0 seconds (migration:455) 2017-08-25 14:22:39,347+0200 INFO (migsrc/ffb71f79) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') starting migration to qemu+tls://microcloud28.rxmgmt.databay.de/system with miguri tcp://172.16.252.128 (migration:480) 2017-08-25 14:22:39,589+0200 ERROR (migsrc/ffb71f79) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') failed to initialize gluster connection (src=0x7fea4c028390 priv=0x7fea4c0263f0): Success (migration:287) 2017-08-25 14:22:39,622+0200 ERROR (migsrc/ffb71f79) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') Failed to migrate (migration:429) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 411, in run self._startUnderlyingMigration(time.time()) File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 487, in _startUnderlyingMigration self._perform_with_conv_schedule(duri, muri) File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 563, in _perform_with_conv_schedule self._perform_migration(duri, muri) File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 529, in _perform_migration self._vm._dom.migrateToURI3(duri, params, flags) File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 69, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 123, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 944, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1939, in migrateToURI3 if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) libvirtError: failed to initialize gluster connection (src=0x7fea4c028390 priv=0x7fea4c0263f0): Success 2017-08-25 14:22:41,012+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:22:41,015+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmIoTunePolicies succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:22:43,525+0200 INFO (periodic/2) [vdsm.api] START repoStats(options=None) from=internal, task_id=0e2c2ad5-89a4-4b4d-89fc-32f7d52b14b9 (api:46) 2017-08-25 14:22:43,525+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000293043', 'lastCheck': '0.5', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000821997', 'lastCheck': '0.5', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000283299', 'lastCheck': '0.6', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000822801', 'lastCheck': '0.5', 'valid': True}} from=internal, task_id=0e2c2ad5-89a4-4b4d-89fc-32f7d52b14b9 (api:52) 2017-08-25 14:22:44,453+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(options=None) from=::ffff:172.16.252.200,56136, flow_id=412cb8de, task_id=4cd8d997-34db-48f6-9ef0-d9834dc833d0 (api:46) 2017-08-25 14:22:44,453+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000293043', 'lastCheck': '1.4', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000821997', 'lastCheck': '1.5', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000283299', 'lastCheck': '1.5', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000822801', 'lastCheck': '1.4', 'valid': True}} from=::ffff:172.16.252.200,56136, flow_id=412cb8de, task_id=4cd8d997-34db-48f6-9ef0-d9834dc833d0 (api:52) 2017-08-25 14:22:44,457+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:22:47,650+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:539) 2017-08-25 14:22:47,996+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getMigrationStatus succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:22:56,016+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:22:56,025+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmIoTunePolicies succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:22:58,574+0200 INFO (periodic/3) [vdsm.api] START repoStats(options=None) from=internal, task_id=c7f2cdfa-7e23-43dd-8f2c-3b203ab145a1 (api:46) 2017-08-25 14:22:58,575+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000291322', 'lastCheck': '0.5', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000735763', 'lastCheck': '3.1', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000295449', 'lastCheck': '3.0', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000783135', 'lastCheck': '3.6', 'valid': True}} from=internal, task_id=c7f2cdfa-7e23-43dd-8f2c-3b203ab145a1 (api:52) 2017-08-25 14:22:59,603+0200 INFO (jsonrpc/1) [vdsm.api] START repoStats(options=None) from=::ffff:172.16.252.200,56136, flow_id=1850b2f6, task_id=82833df7-375a-477a-a4c7-7b67ef11dd23 (api:46) 2017-08-25 14:22:59,603+0200 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000291322', 'lastCheck': '1.5', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000735763', 'lastCheck': '4.1', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000295449', 'lastCheck': '4.1', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000783135', 'lastCheck': '4.6', 'valid': True}} from=::ffff:172.16.252.200,56136, flow_id=1850b2f6, task_id=82833df7-375a-477a-a4c7-7b67ef11dd23 (api:52) 2017-08-25 14:22:59,607+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:23:02,970+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:539) 2017-08-25 14:23:10,643+0200 INFO (jsonrpc/6) [vdsm.api] START migrate(params={u'incomingLimit': 2, u'src': u'microcloud22.rxmgmt.databay.de', u'dstqemu': u'172.16.252.121', u'autoConverge': u'true', u'tunneled': u'false', u'enableGuestEvents': True, u'dst': u'microcloud21.rxmgmt.databay.de:54321', u'convergenceSchedule': {u'init': [{u'params': [u'100'], u'name': u'setDowntime'}], u'stalling': [{u'action': {u'params': [u'150'], u'name': u'setDowntime'}, u'limit': 1}, {u'action': {u'params': [u'200'], u'name': u'setDowntime'}, u'limit': 2}, {u'action': {u'params': [u'300'], u'name': u'setDowntime'}, u'limit': 3}, {u'action': {u'params': [u'400'], u'name': u'setDowntime'}, u'limit': 4}, {u'action': {u'params': [u'500'], u'name': u'setDowntime'}, u'limit': 6}, {u'action': {u'params': [], u'name': u'abort'}, u'limit': -1}]}, u'vmId': u'ffb71f79-54cd-4f0e-b6b5-3670236cb497', u'abortOnError': u'true', u'outgoingLimit': 2, u'compressed': u'false', u'method': u'online', 'mode': 'remote'}) from=::ffff:172.16.252.200,56136, flow_id=1c92b857-14f8-49f4-9e09-f1afef2b608c (api:46) 2017-08-25 14:23:10,646+0200 INFO (jsonrpc/6) [vdsm.api] FINISH migrate return={'status': {'message': 'Migration in progress', 'code': 0}, 'progress': 0} from=::ffff:172.16.252.200,56136, flow_id=1c92b857-14f8-49f4-9e09-f1afef2b608c (api:52) 2017-08-25 14:23:10,646+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.migrate succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:23:11,030+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:539) 2017-08-25 14:23:11,033+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmIoTunePolicies succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:23:11,307+0200 INFO (migsrc/ffb71f79) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') Creation of destination VM took: 0 seconds (migration:455) 2017-08-25 14:23:11,307+0200 INFO (migsrc/ffb71f79) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') starting migration to qemu+tls://microcloud21.rxmgmt.databay.de/system with miguri tcp://172.16.252.121 (migration:480) 2017-08-25 14:23:11,944+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID=u'5d99af76-33b5-47d8-99da-1f32413c7bb0', spUUID=u'00000001-0001-0001-0001-0000000000b9', imgUUID=u'9c007b27-0ab7-4474-9317-a294fd04c65f', volUUID=u'ba2dfe7f-bdcf-4680-878d-7d71272f3ffb', options=None) from=internal, task_id=cc787992-063a-4eea-aff7-13454b483f29 (api:46) 2017-08-25 14:23:11,946+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '8133296128', 'apparentsize': '10737418240'} from=internal, task_id=cc787992-063a-4eea-aff7-13454b483f29 (api:52) 2017-08-25 14:23:12,586+0200 ERROR (migsrc/ffb71f79) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') failed to initialize gluster connection (src=0x7f30d401e5f0 priv=0x7f30d4002e30): Invalid argument (migration:287) 2017-08-25 14:23:12,627+0200 ERROR (migsrc/ffb71f79) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') Failed to migrate (migration:429) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 411, in run self._startUnderlyingMigration(time.time()) File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 487, in _startUnderlyingMigration self._perform_with_conv_schedule(duri, muri) File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 563, in _perform_with_conv_schedule self._perform_migration(duri, muri) File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 529, in _perform_migration self._vm._dom.migrateToURI3(duri, params, flags) File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 69, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 123, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 944, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1939, in migrateToURI3 if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) libvirtError: failed to initialize gluster connection (src=0x7f30d401e5f0 priv=0x7f30d4002e30): Invalid argument 2017-08-25 14:23:13,625+0200 INFO (periodic/0) [vdsm.api] START repoStats(options=None) from=internal, task_id=d89d11bb-8ae1-4fa1-b54c-dcbf1c57db57 (api:46) 2017-08-25 14:23:13,626+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000304122', 'lastCheck': '0.6', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000765793', 'lastCheck': '0.6', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000329078', 'lastCheck': '0.7', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00076071', 'lastCheck': '0.6', 'valid': True}} from=internal, task_id=d89d11bb-8ae1-4fa1-b54c-dcbf1c57db57 (api:52) 2017-08-25 14:23:15,014+0200 INFO (itmap/0) [IOProcessClient] Starting client ioprocess-1196 (__init__:330) 2017-08-25 14:23:15,021+0200 INFO (ioprocess/15817) [IOProcess] Starting ioprocess (__init__:452) 2017-08-25 14:23:15,021+0200 INFO (itmap/1) [IOProcessClient] Starting client ioprocess-1197 (__init__:330) 2017-08-25 14:23:15,029+0200 INFO (ioprocess/15825) [IOProcess] Starting ioprocess (__init__:452) 2017-08-25 14:23:15,034+0200 INFO (monitor/a7fbaaa) [storage.StorageDomain] Resource namespace 01_img_a7fbaaad-7043-4391-9523-3bedcdc4fb0d already registered (sd:727) 2017-08-25 14:23:15,034+0200 INFO (monitor/a7fbaaa) [storage.StorageDomain] Resource namespace 02_vol_a7fbaaad-7043-4391-9523-3bedcdc4fb0d already registered (sd:736) 2017-08-25 14:23:15,507+0200 INFO (monitor/2b2a44f) [storage.StorageDomain] Resource namespace 01_img_2b2a44fc-f2bd-47cd-b7af-00be59e30a35 already registered (sd:727) 2017-08-25 14:23:15,507+0200 INFO (monitor/2b2a44f) [storage.StorageDomain] Resource namespace 02_vol_2b2a44fc-f2bd-47cd-b7af-00be59e30a35 already registered (sd:736) 2017-08-25 14:23:15,788+0200 INFO (jsonrpc/4) [vdsm.api] START repoStats(options=None) from=::ffff:172.16.252.200,56136, flow_id=6cfe2b98, task_id=28ea2ca8-9515-49d3-be4e-dc883c9ca381 (api:46) 2017-08-25 14:23:15,788+0200 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000304122', 'lastCheck': '2.8', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000765793', 'lastCheck': '0.3', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000329078', 'lastCheck': '0.2', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00076071', 'lastCheck': '0.7', 'valid': True}} from=::ffff:172.16.252.200,56136, flow_id=6cfe2b98, task_id=28ea2ca8-9515-49d3-be4e-dc883c9ca381 (api:52) 2017-08-25 14:23:15,791+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:539) 2017-08-25 14:23:18,194+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:23:18,506+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call VM.getMigrationStatus succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:23:26,046+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:23:26,049+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmIoTunePolicies succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:23:28,692+0200 INFO (periodic/3) [vdsm.api] START repoStats(options=None) from=internal, task_id=0cf6a2ab-d11e-421d-b705-d1e617a473e1 (api:46) 2017-08-25 14:23:28,692+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000298914', 'lastCheck': '0.6', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000716162', 'lastCheck': '3.2', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000272715', 'lastCheck': '3.1', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000637443', 'lastCheck': '3.6', 'valid': True}} from=internal, task_id=0cf6a2ab-d11e-421d-b705-d1e617a473e1 (api:52) 2017-08-25 14:23:30,949+0200 INFO (jsonrpc/6) [vdsm.api] START repoStats(options=None) from=::ffff:172.16.252.200,56136, flow_id=4bf20ec7, task_id=1a2b199a-0dfd-4335-a228-e2de8a23beba (api:46) 2017-08-25 14:23:30,949+0200 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000298914', 'lastCheck': '2.9', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000716162', 'lastCheck': '5.4', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000272715', 'lastCheck': '5.4', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000637443', 'lastCheck': '5.9', 'valid': True}} from=::ffff:172.16.252.200,56136, flow_id=4bf20ec7, task_id=1a2b199a-0dfd-4335-a228-e2de8a23beba (api:52) 2017-08-25 14:23:30,953+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:539) 2017-08-25 14:23:33,506+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:23:41,049+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:23:41,067+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmIoTunePolicies succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:23:43,762+0200 INFO (periodic/3) [vdsm.api] START repoStats(options=None) from=internal, task_id=b78f2532-1b18-47d5-a5cc-a8f39d296ee7 (api:46) 2017-08-25 14:23:43,762+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000296471', 'lastCheck': '0.8', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000681868', 'lastCheck': '0.8', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000274172', 'lastCheck': '0.8', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000784273', 'lastCheck': '0.7', 'valid': True}} from=internal, task_id=b78f2532-1b18-47d5-a5cc-a8f39d296ee7 (api:52) 2017-08-25 14:23:47,377+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(options=None) from=::ffff:172.16.252.200,56136, flow_id=7b0b9ae4, task_id=0b0f9c2d-b7aa-4e71-9c0a-062c30cc20cf (api:46) 2017-08-25 14:23:47,377+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000296471', 'lastCheck': '4.4', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000681868', 'lastCheck': '1.9', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000274172', 'lastCheck': '1.8', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000784273', 'lastCheck': '2.3', 'valid': True}} from=::ffff:172.16.252.200,56136, flow_id=7b0b9ae4, task_id=0b0f9c2d-b7aa-4e71-9c0a-062c30cc20cf (api:52) 2017-08-25 14:23:47,382+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:539) 2017-08-25 14:23:48,823+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:23:56,064+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:23:56,076+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmIoTunePolicies succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:23:58,820+0200 INFO (periodic/2) [vdsm.api] START repoStats(options=None) from=internal, task_id=80747589-bb6c-47e5-99f3-85621725b8f6 (api:46) 2017-08-25 14:23:58,820+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000302304', 'lastCheck': '0.7', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00072861', 'lastCheck': '3.3', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000341366', 'lastCheck': '3.3', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000759674', 'lastCheck': '3.8', 'valid': True}} from=internal, task_id=80747589-bb6c-47e5-99f3-85621725b8f6 (api:52) 2017-08-25 14:24:02,658+0200 INFO (jsonrpc/6) [vdsm.api] START repoStats(options=None) from=::ffff:172.16.252.200,56136, flow_id=22bf59a8, task_id=6023a6eb-e670-45c2-8ae5-ecd7c6464094 (api:46) 2017-08-25 14:24:02,658+0200 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000302304', 'lastCheck': '4.6', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00072861', 'lastCheck': '7.1', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000341366', 'lastCheck': '7.1', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000759674', 'lastCheck': '7.6', 'valid': True}} from=::ffff:172.16.252.200,56136, flow_id=22bf59a8, task_id=6023a6eb-e670-45c2-8ae5-ecd7c6464094 (api:52) 2017-08-25 14:24:02,662+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:539) 2017-08-25 14:24:03,989+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:24:11,081+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:24:11,084+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmIoTunePolicies succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:24:11,947+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID=u'5d99af76-33b5-47d8-99da-1f32413c7bb0', spUUID=u'00000001-0001-0001-0001-0000000000b9', imgUUID=u'9c007b27-0ab7-4474-9317-a294fd04c65f', volUUID=u'ba2dfe7f-bdcf-4680-878d-7d71272f3ffb', options=None) from=internal, task_id=d2aa9f47-9795-4fb3-a0e7-437db9a53ade (api:46) 2017-08-25 14:24:11,951+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '8133296128', 'apparentsize': '10737418240'} from=internal, task_id=d2aa9f47-9795-4fb3-a0e7-437db9a53ade (api:52) 2017-08-25 14:24:13,900+0200 INFO (periodic/0) [vdsm.api] START repoStats(options=None) from=internal, task_id=1f882a0d-7774-4f17-86ec-f3106fcb82e3 (api:46) 2017-08-25 14:24:13,900+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000300742', 'lastCheck': '0.9', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000744307', 'lastCheck': '0.9', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000309006', 'lastCheck': '0.9', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000675401', 'lastCheck': '0.8', 'valid': True}} from=internal, task_id=1f882a0d-7774-4f17-86ec-f3106fcb82e3 (api:52) 2017-08-25 14:24:15,533+0200 INFO (monitor/2b2a44f) [IOProcessClient] Closing client ioprocess-1195 (__init__:598) 2017-08-25 14:24:15,533+0200 INFO (monitor/2b2a44f) [IOProcessClient] Closing client ioprocess-1196 (__init__:598) 2017-08-25 14:24:15,533+0200 INFO (monitor/2b2a44f) [IOProcessClient] Closing client ioprocess-1194 (__init__:598) 2017-08-25 14:24:15,534+0200 INFO (monitor/2b2a44f) [IOProcessClient] Closing client ioprocess-1197 (__init__:598) 2017-08-25 14:24:17,901+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(options=None) from=::ffff:172.16.252.200,56136, flow_id=7cf25c35, task_id=6b9869f3-93f2-4f13-97ee-dc0f481f7dfb (api:46) 2017-08-25 14:24:17,901+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000300742', 'lastCheck': '4.9', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000744307', 'lastCheck': '2.4', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000309006', 'lastCheck': '2.3', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000675401', 'lastCheck': '2.8', 'valid': True}} from=::ffff:172.16.252.200,56136, flow_id=7cf25c35, task_id=6b9869f3-93f2-4f13-97ee-dc0f481f7dfb (api:52) 2017-08-25 14:24:17,905+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:24:20,156+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:24:20,570+0200 INFO (jsonrpc/1) [vdsm.api] START migrate(params={u'incomingLimit': 2, u'src': u'microcloud22.rxmgmt.databay.de', u'dstqemu': u'172.16.252.123', u'autoConverge': u'true', u'tunneled': u'false', u'enableGuestEvents': True, u'dst': u'microcloud23.rxmgmt.databay.de:54321', u'convergenceSchedule': {u'init': [{u'params': [u'100'], u'name': u'setDowntime'}], u'stalling': [{u'action': {u'params': [u'150'], u'name': u'setDowntime'}, u'limit': 1}, {u'action': {u'params': [u'200'], u'name': u'setDowntime'}, u'limit': 2}, {u'action': {u'params': [u'300'], u'name': u'setDowntime'}, u'limit': 3}, {u'action': {u'params': [u'400'], u'name': u'setDowntime'}, u'limit': 4}, {u'action': {u'params': [u'500'], u'name': u'setDowntime'}, u'limit': 6}, {u'action': {u'params': [], u'name': u'abort'}, u'limit': -1}]}, u'vmId': u'ffb71f79-54cd-4f0e-b6b5-3670236cb497', u'abortOnError': u'true', u'outgoingLimit': 2, u'compressed': u'false', u'method': u'online', 'mode': 'remote'}) from=::ffff:172.16.252.200,56136, flow_id=618f4942-952e-4108-8003-1a61869ec683 (api:46) 2017-08-25 14:24:20,573+0200 INFO (jsonrpc/1) [vdsm.api] FINISH migrate return={'status': {'message': 'Migration in progress', 'code': 0}, 'progress': 0} from=::ffff:172.16.252.200,56136, flow_id=618f4942-952e-4108-8003-1a61869ec683 (api:52) 2017-08-25 14:24:20,573+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call VM.migrate succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:24:21,017+0200 INFO (migsrc/ffb71f79) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') Creation of destination VM took: 0 seconds (migration:455) 2017-08-25 14:24:21,017+0200 INFO (migsrc/ffb71f79) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') starting migration to qemu+tls://microcloud23.rxmgmt.databay.de/system with miguri tcp://172.16.252.123 (migration:480) 2017-08-25 14:24:21,230+0200 ERROR (migsrc/ffb71f79) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') failed to initialize gluster connection (src=0x7f740c01f900 priv=0x7f740c0044c0): Success (migration:287) 2017-08-25 14:24:21,266+0200 ERROR (migsrc/ffb71f79) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') Failed to migrate (migration:429) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 411, in run self._startUnderlyingMigration(time.time()) File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 487, in _startUnderlyingMigration self._perform_with_conv_schedule(duri, muri) File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 563, in _perform_with_conv_schedule self._perform_migration(duri, muri) File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 529, in _perform_migration self._vm._dom.migrateToURI3(duri, params, flags) File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 69, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 123, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 944, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1939, in migrateToURI3 if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) libvirtError: failed to initialize gluster connection (src=0x7f740c01f900 priv=0x7f740c0044c0): Success 2017-08-25 14:24:26,093+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:24:26,095+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmIoTunePolicies succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:24:28,946+0200 INFO (periodic/2) [vdsm.api] START repoStats(options=None) from=internal, task_id=ae40ed24-56f4-44c1-8688-86e9ce91fa38 (api:46) 2017-08-25 14:24:28,946+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.0003044', 'lastCheck': '0.9', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000717384', 'lastCheck': '3.4', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000292725', 'lastCheck': '3.4', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000674707', 'lastCheck': '3.9', 'valid': True}} from=internal, task_id=ae40ed24-56f4-44c1-8688-86e9ce91fa38 (api:52) 2017-08-25 14:24:33,992+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(options=None) from=::ffff:172.16.252.200,56136, flow_id=58c5ace5, task_id=f137f2c9-9913-4f0b-a445-1a284f28c0f1 (api:46) 2017-08-25 14:24:33,992+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000293336', 'lastCheck': '1.0', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000675592', 'lastCheck': '1.0', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000278958', 'lastCheck': '1.0', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000776874', 'lastCheck': '0.9', 'valid': True}} from=::ffff:172.16.252.200,56136, flow_id=58c5ace5, task_id=f137f2c9-9913-4f0b-a445-1a284f28c0f1 (api:52) 2017-08-25 14:24:33,996+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:24:35,572+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:24:37,093+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getMigrationStatus succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:24:41,098+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:24:41,100+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmIoTunePolicies succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:24:44,027+0200 INFO (periodic/2) [vdsm.api] START repoStats(options=None) from=internal, task_id=075b4496-adc4-4770-b056-df4d1ca37c5d (api:46) 2017-08-25 14:24:44,027+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000332244', 'lastCheck': '1.0', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000760547', 'lastCheck': '1.0', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000329424', 'lastCheck': '1.1', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000717052', 'lastCheck': '1.0', 'valid': True}} from=internal, task_id=075b4496-adc4-4770-b056-df4d1ca37c5d (api:52) 2017-08-25 14:24:49,151+0200 INFO (jsonrpc/1) [vdsm.api] START repoStats(options=None) from=::ffff:172.16.252.200,56136, flow_id=18fe680d, task_id=311a766f-ca7d-4802-a46c-2f8ab317c914 (api:46) 2017-08-25 14:24:49,151+0200 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000332244', 'lastCheck': '1.1', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000760547', 'lastCheck': '3.6', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000329424', 'lastCheck': '3.6', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000717052', 'lastCheck': '4.1', 'valid': True}} from=::ffff:172.16.252.200,56136, flow_id=18fe680d, task_id=311a766f-ca7d-4802-a46c-2f8ab317c914 (api:52) 2017-08-25 14:24:49,155+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:24:52,071+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:24:54,632+0200 INFO (jsonrpc/6) [vdsm.api] START migrate(params={u'incomingLimit': 2, u'src': u'microcloud22.rxmgmt.databay.de', u'dstqemu': u'172.16.252.125', u'autoConverge': u'true', u'tunneled': u'false', u'enableGuestEvents': True, u'dst': u'microcloud25.rxmgmt.databay.de:54321', u'convergenceSchedule': {u'init': [{u'params': [u'100'], u'name': u'setDowntime'}], u'stalling': [{u'action': {u'params': [u'150'], u'name': u'setDowntime'}, u'limit': 1}, {u'action': {u'params': [u'200'], u'name': u'setDowntime'}, u'limit': 2}, {u'action': {u'params': [u'300'], u'name': u'setDowntime'}, u'limit': 3}, {u'action': {u'params': [u'400'], u'name': u'setDowntime'}, u'limit': 4}, {u'action': {u'params': [u'500'], u'name': u'setDowntime'}, u'limit': 6}, {u'action': {u'params': [], u'name': u'abort'}, u'limit': -1}]}, u'vmId': u'ffb71f79-54cd-4f0e-b6b5-3670236cb497', u'abortOnError': u'true', u'outgoingLimit': 2, u'compressed': u'false', u'method': u'online', 'mode': 'remote'}) from=::ffff:172.16.252.200,56136, flow_id=1a2564b5-b739-47aa-b9dd-edda1c41b477 (api:46) 2017-08-25 14:24:54,635+0200 INFO (jsonrpc/6) [vdsm.api] FINISH migrate return={'status': {'message': 'Migration in progress', 'code': 0}, 'progress': 0} from=::ffff:172.16.252.200,56136, flow_id=1a2564b5-b739-47aa-b9dd-edda1c41b477 (api:52) 2017-08-25 14:24:54,635+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.migrate succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:24:55,034+0200 INFO (migsrc/ffb71f79) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') Creation of destination VM took: 0 seconds (migration:455) 2017-08-25 14:24:55,034+0200 INFO (migsrc/ffb71f79) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') starting migration to qemu+tls://microcloud25.rxmgmt.databay.de/system with miguri tcp://172.16.252.125 (migration:480) 2017-08-25 14:24:56,117+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:24:56,119+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmIoTunePolicies succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:24:59,046+0200 INFO (periodic/3) [vdsm.api] START repoStats(options=None) from=internal, task_id=fa1a8571-227a-4946-ac03-accd9c4776f6 (api:46) 2017-08-25 14:24:59,046+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000294938', 'lastCheck': '1.0', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000748909', 'lastCheck': '3.5', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000302267', 'lastCheck': '3.5', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00070846', 'lastCheck': '4.0', 'valid': True}} from=internal, task_id=fa1a8571-227a-4946-ac03-accd9c4776f6 (api:52) 2017-08-25 14:25:04,412+0200 INFO (jsonrpc/4) [vdsm.api] START repoStats(options=None) from=::ffff:172.16.252.200,56136, flow_id=7a4178c9, task_id=a540635c-581d-4dca-a06e-69bc67421e66 (api:46) 2017-08-25 14:25:04,412+0200 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000270619', 'lastCheck': '1.4', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000675603', 'lastCheck': '1.4', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000425784', 'lastCheck': '1.4', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00504999', 'lastCheck': '1.4', 'valid': True}} from=::ffff:172.16.252.200,56136, flow_id=7a4178c9, task_id=a540635c-581d-4dca-a06e-69bc67421e66 (api:52) 2017-08-25 14:25:04,416+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:25:05,036+0200 INFO (migmon/ffb71f79) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') Migration Progress: 10 seconds elapsed, 29% of data processed, total data: 2121MB, processed data: 447MB, remaining data: 1517MB, transfer speed 51MBps, zero pages: 40205MB, compressed: 0MB, dirty rate: 0, memory iteration: 1 (migration:815) 2017-08-25 14:25:07,358+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:25:11,132+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:25:11,950+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID=u'5d99af76-33b5-47d8-99da-1f32413c7bb0', spUUID=u'00000001-0001-0001-0001-0000000000b9', imgUUID=u'9c007b27-0ab7-4474-9317-a294fd04c65f', volUUID=u'ba2dfe7f-bdcf-4680-878d-7d71272f3ffb', options=None) from=internal, task_id=14bff9c0-e25c-44ac-8341-f7a9e5ef454a (api:46) 2017-08-25 14:25:11,953+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '8133296128', 'apparentsize': '10737418240'} from=internal, task_id=14bff9c0-e25c-44ac-8341-f7a9e5ef454a (api:52) 2017-08-25 14:25:14,111+0200 INFO (periodic/0) [vdsm.api] START repoStats(options=None) from=internal, task_id=950ddead-9fd8-4246-95a4-2cc6bfe39d89 (api:46) 2017-08-25 14:25:14,111+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000298554', 'lastCheck': '1.1', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.0007038', 'lastCheck': '1.1', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000403163', 'lastCheck': '1.2', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00506299', 'lastCheck': '1.1', 'valid': True}} from=internal, task_id=950ddead-9fd8-4246-95a4-2cc6bfe39d89 (api:52) 2017-08-25 14:25:15,038+0200 INFO (migmon/ffb71f79) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') Migration Progress: 20 seconds elapsed, 54% of data processed, total data: 2121MB, processed data: 967MB, remaining data: 990MB, transfer speed 52MBps, zero pages: 42440MB, compressed: 0MB, dirty rate: 0, memory iteration: 1 (migration:815) 2017-08-25 14:25:20,254+0200 INFO (jsonrpc/1) [vdsm.api] START repoStats(options=None) from=::ffff:172.16.252.200,56136, flow_id=71a1c479, task_id=4f0cca90-bbd6-4fb8-9e98-ce8b7c426eb1 (api:46) 2017-08-25 14:25:20,254+0200 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000298554', 'lastCheck': '2.2', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.0007038', 'lastCheck': '4.7', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000403163', 'lastCheck': '4.7', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00506299', 'lastCheck': '5.2', 'valid': True}} from=::ffff:172.16.252.200,56136, flow_id=71a1c479, task_id=4f0cca90-bbd6-4fb8-9e98-ce8b7c426eb1 (api:52) 2017-08-25 14:25:20,258+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:25:22,938+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:25:25,039+0200 INFO (migmon/ffb71f79) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') Migration Progress: 30 seconds elapsed, 91% of data processed, total data: 2121MB, processed data: 1487MB, remaining data: 200MB, transfer speed 52MBps, zero pages: 111708MB, compressed: 0MB, dirty rate: 0, memory iteration: 1 (migration:815) 2017-08-25 14:25:26,151+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:25:29,113+0200 INFO (libvirt/events) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') CPU stopped: onSuspend (vm:5048) 2017-08-25 14:25:29,139+0200 INFO (periodic/3) [vdsm.api] START repoStats(options=None) from=internal, task_id=0cc66cf5-a521-4e18-a555-7a068cc626df (api:46) 2017-08-25 14:25:29,139+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000287577', 'lastCheck': '1.0', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000684865', 'lastCheck': '3.6', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000472416', 'lastCheck': '3.5', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000728642', 'lastCheck': '4.0', 'valid': True}} from=internal, task_id=0cc66cf5-a521-4e18-a555-7a068cc626df (api:52) 2017-08-25 14:25:31,017+0200 INFO (migsrc/ffb71f79) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') migration took 36 seconds to complete (migration:492) 2017-08-25 14:25:31,018+0200 INFO (migsrc/ffb71f79) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') Changed state to Down: Migration succeeded (code=4) (vm:1221) 2017-08-25 14:25:31,019+0200 INFO (migsrc/ffb71f79) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') Stopping connection (guestagent:430) 2017-08-25 14:25:32,059+0200 INFO (jsonrpc/3) [vdsm.api] START destroy(gracefulAttempts=1) from=::ffff:172.16.252.200,56136 (api:46) 2017-08-25 14:25:32,060+0200 INFO (jsonrpc/3) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') Release VM resources (vm:4254) 2017-08-25 14:25:32,060+0200 WARN (jsonrpc/3) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') trying to set state to Powering down when already Down (vm:361) 2017-08-25 14:25:32,060+0200 INFO (jsonrpc/3) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') Stopping connection (guestagent:430) 2017-08-25 14:25:32,060+0200 INFO (jsonrpc/3) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') _destroyVmGraceful attempt #0 (vm:4290) 2017-08-25 14:25:32,060+0200 INFO (jsonrpc/3) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') VM 'ffb71f79-54cd-4f0e-b6b5-3670236cb497' already down and destroyed (vm:4308) 2017-08-25 14:25:32,061+0200 INFO (jsonrpc/3) [vdsm.api] START teardownImage(sdUUID=u'5d99af76-33b5-47d8-99da-1f32413c7bb0', spUUID=u'00000001-0001-0001-0001-0000000000b9', imgUUID=u'9c007b27-0ab7-4474-9317-a294fd04c65f', volUUID=None) from=::ffff:172.16.252.200,56136, task_id=103b48e7-48ce-4c82-9c59-96bd0dbe1c0c (api:46) 2017-08-25 14:25:32,061+0200 INFO (jsonrpc/3) [vdsm.api] FINISH teardownImage return=None from=::ffff:172.16.252.200,56136, task_id=103b48e7-48ce-4c82-9c59-96bd0dbe1c0c (api:52) 2017-08-25 14:25:32,062+0200 INFO (jsonrpc/3) [virt.vm] (vmId='ffb71f79-54cd-4f0e-b6b5-3670236cb497') Stopping connection (guestagent:430) 2017-08-25 14:25:32,062+0200 WARN (jsonrpc/3) [root] File: /var/lib/libvirt/qemu/channels/ffb71f79-54cd-4f0e-b6b5-3670236cb497.com.redhat.rhevm.vdsm already removed (utils:120) 2017-08-25 14:25:32,062+0200 WARN (jsonrpc/3) [root] File: /var/lib/libvirt/qemu/channels/ffb71f79-54cd-4f0e-b6b5-3670236cb497.org.qemu.guest_agent.0 already removed (utils:120) 2017-08-25 14:25:32,062+0200 INFO (jsonrpc/3) [vdsm.api] START inappropriateDevices(thiefId=u'ffb71f79-54cd-4f0e-b6b5-3670236cb497') from=::ffff:172.16.252.200,56136, task_id=9b3de392-e063-41ec-aa9e-396c28d1696e (api:46) 2017-08-25 14:25:32,063+0200 INFO (jsonrpc/3) [vdsm.api] FINISH inappropriateDevices return=None from=::ffff:172.16.252.200,56136, task_id=9b3de392-e063-41ec-aa9e-396c28d1696e (api:52) 2017-08-25 14:25:32,083+0200 INFO (jsonrpc/3) [root] (hooks:108) 2017-08-25 14:25:32,084+0200 INFO (jsonrpc/3) [vdsm.api] FINISH destroy return={'status': {'message': 'Done', 'code': 0}} from=::ffff:172.16.252.200,56136 (api:52) 2017-08-25 14:25:32,084+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call VM.destroy succeeded in 0.03 seconds (__init__:539) 2017-08-25 14:25:35,535+0200 INFO (jsonrpc/0) [vdsm.api] START repoStats(options=None) from=::ffff:172.16.252.200,56136, flow_id=68d2aa42, task_id=01b34549-fdbc-4100-bbc5-99e73a5929ea (api:46) 2017-08-25 14:25:35,535+0200 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={u'7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000307689', 'lastCheck': '2.5', 'valid': True}, u'2b2a44fc-f2bd-47cd-b7af-00be59e30a35': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000756093', 'lastCheck': '2.5', 'valid': True}, u'5d99af76-33b5-47d8-99da-1f32413c7bb0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000373814', 'lastCheck': '2.6', 'valid': True}, u'a7fbaaad-7043-4391-9523-3bedcdc4fb0d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000640381', 'lastCheck': '0.4', 'valid': True}} from=::ffff:172.16.252.200,56136, flow_id=68d2aa42, task_id=01b34549-fdbc-4100-bbc5-99e73a5929ea (api:52) 2017-08-25 14:25:35,538+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.00 seconds (__init__:539) 2017-08-25 14:25:38,182+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539)