<html><head></head><body><div style="color:#000; background-color:#fff; font-family:Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:10px"><div id="yui_3_16_0_ym19_1_1519085807369_70843">Sorry can't attached log file it's too big file<br></div><div id="yui_3_16_0_ym19_1_1519085807369_70846"><br></div><div><br></div><div>VDSM.log for node 1</div><div id="yui_3_16_0_ym19_1_1519085807369_70844"><br></div><div id="yui_3_16_0_ym19_1_1519085807369_70530" dir="ltr">2766', 'lastCheck': '4.9', 'valid': True}} from=internal, task_id=645d456e-f59f-4b1c-9e97-fc82d19a36b1 (api:52)<br id="yui_3_16_0_ym19_1_1519085807369_70565">2018-02-20 14:38:47,222+0800 INFO (jsonrpc/3) [vdsm.api] START repoStats(options=None) from=::ffff:10.10.43.1,60554, flow_id=3b2e802e, task_id=28c795d1-1639-4e68-a9fe-a00006be268f (api:46)<br id="yui_3_16_0_ym19_1_1519085807369_70566">2018-02-20 14:38:47,222+0800 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={'f3e372e3-1251-4195-a4b9-1027e40059df': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000119109', 'lastCheck': '6.0', 'valid': True}, 'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000195263', 'lastCheck': '6.0', 'valid': True}, '65ca2e2d-b472-4bee-85b4-09a161464b20': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00022988', 'lastCheck': '6.0', 'valid': True}, '4bf2ba2f-f57a-4d9f-b42a-fb78f440a358': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000270206', 'lastCheck': '5.9', 'valid': True}, '42e591b7-f86c-4b67-a3d2-40cc007f7662': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000272766', 'lastCheck': '6.0', 'valid': True}} from=::ffff:10.10.43.1,60554, flow_id=3b2e802e, task_id=28c795d1-1639-4e68-a9fe-a00006be268f (api:52)<br id="yui_3_16_0_ym19_1_1519085807369_70567">2018-02-20 14:38:47,226+0800 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.00 seconds (__init__:539)<br id="yui_3_16_0_ym19_1_1519085807369_70568">2018-02-20 14:38:55,252+0800 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539)<br id="yui_3_16_0_ym19_1_1519085807369_70569">2018-02-20 14:38:58,566+0800 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539)<br id="yui_3_16_0_ym19_1_1519085807369_70570">2018-02-20 14:39:01,093+0800 INFO (periodic/1) [vdsm.api] START repoStats(options=None) from=internal, task_id=04f55ded-5841-44ae-a376-4f6e723e4b10 (api:46)<br id="yui_3_16_0_ym19_1_1519085807369_70571">2018-02-20 14:39:01,093+0800 INFO (periodic/1) [vdsm.api] FINISH repoStats return={'f3e372e3-1251-4195-a4b9-1027e40059df': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '9.3888e-05', 'lastCheck': '9.9', 'valid': True}, 'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000197535', 'lastCheck': '9.8', 'valid': True}, '65ca2e2d-b472-4bee-85b4-09a161464b20': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000191456', 'lastCheck': '9.8', 'valid': True}, '4bf2ba2f-f57a-4d9f-b42a-fb78f440a358': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00026365', 'lastCheck': '9.8', 'valid': True}, '42e591b7-f86c-4b67-a3d2-40cc007f7662': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000243658', 'lastCheck': '0.0', 'valid': True}} from=internal, task_id=04f55ded-5841-44ae-a376-4f6e723e4b10 (api:52)<br id="yui_3_16_0_ym19_1_1519085807369_70572">2018-02-20 14:39:02,295+0800 INFO (jsonrpc/6) [vdsm.api] START repoStats(options=None) from=::ffff:10.10.43.1,60554, flow_id=13149d6e, task_id=5f19d76a-d343-4dc4-ad09-024ec27f7443 (api:46)<br id="yui_3_16_0_ym19_1_1519085807369_70573">2018-02-20 14:39:02,295+0800 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'f3e372e3-1251-4195-a4b9-1027e40059df': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '9.4759e-05', 'lastCheck': '1.1', 'valid': True}, 'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000183158', 'lastCheck': '1.0', 'valid': True}, '65ca2e2d-b472-4bee-85b4-09a161464b20': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000222609', 'lastCheck': '1.0', 'valid': True}, '4bf2ba2f-f57a-4d9f-b42a-fb78f440a358': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000211253', 'lastCheck': '1.0', 'valid': True}, '42e591b7-f86c-4b67-a3d2-40cc007f7662': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000243658', 'lastCheck': '1.1', 'valid': True}} from=::ffff:10.10.43.1,60554, flow_id=13149d6e, task_id=5f19d76a-d343-4dc4-ad09-024ec27f7443 (api:52)<br id="yui_3_16_0_ym19_1_1519085807369_70574">2018-02-20 14:39:02,300+0800 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:539)<br id="yui_3_16_0_ym19_1_1519085807369_70575">2018-02-20 14:39:10,270+0800 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539)<br id="yui_3_16_0_ym19_1_1519085807369_70576">2018-02-20 14:39:13,631+0800 INFO (jsonrpc/0) [vdsm.api] START connectStorageServer(domType=1, spUUID=u'00000000-0000-0000-0000-000000000000', conList=[{u'id': u'5e4c94f1-f3b9-4fbd-a6c7-e732d0fe3123', u'connection': u'dev2node1.lares.com.ph:/run/media/root/Slave1Data/dataNode1', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'protocol_version': u'auto', u'password': '********', u'port': u''}], options=None) from=::ffff:10.10.43.1,60554, flow_id=15b57417, task_id=bfce1e70-ef4a-4e13-aaaa-7a66aaf44429 (api:46)<br id="yui_3_16_0_ym19_1_1519085807369_70577">2018-02-20 14:39:13,633+0800 INFO (jsonrpc/0) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'status': 0, 'id': u'5e4c94f1-f3b9-4fbd-a6c7-e732d0fe3123'}]} from=::ffff:10.10.43.1,60554, flow_id=15b57417, task_id=bfce1e70-ef4a-4e13-aaaa-7a66aaf44429 (api:52)<br id="yui_3_16_0_ym19_1_1519085807369_70578">2018-02-20 14:39:13,634+0800 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer succeeded in 0.00 seconds (__init__:539)<br id="yui_3_16_0_ym19_1_1519085807369_70579">2018-02-20 14:39:13,806+0800 INFO (jsonrpc/7) [vdsm.api] START connectStoragePool(spUUID=u'5a865884-0366-0330-02b8-0000000002d4', hostID=1, msdUUID=u'f3e372e3-1251-4195-a4b9-1027e40059df', masterVersion=65, domainsMap={u'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': u'active', u'4bf2ba2f-f57a-4d9f-b42a-fb78f440a358': u'active', u'225e1975-8121-4370-b317-86e964ae326f': u'attached', u'f3e372e3-1251-4195-a4b9-1027e40059df': u'active', u'65ca2e2d-b472-4bee-85b4-09a161464b20': u'active', u'42e591b7-f86c-4b67-a3d2-40cc007f7662': u'active'}, options=None) from=::ffff:10.10.43.1,60554, flow_id=15b57417, task_id=927a5d9a-4304-4776-b5c9-22ba5d0cb853 (api:46)<br id="yui_3_16_0_ym19_1_1519085807369_70580">2018-02-20 14:39:13,807+0800 INFO (jsonrpc/7) [storage.StoragePoolMemoryBackend] new storage pool master version 65 and domains map {u'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': u'Active', u'4bf2ba2f-f57a-4d9f-b42a-fb78f440a358': u'Active', u'225e1975-8121-4370-b317-86e964ae326f': u'Attached', u'f3e372e3-1251-4195-a4b9-1027e40059df': u'Active', u'65ca2e2d-b472-4bee-85b4-09a161464b20': u'Active', u'42e591b7-f86c-4b67-a3d2-40cc007f7662': u'Active'} (spbackends:450)<br id="yui_3_16_0_ym19_1_1519085807369_70581"></div><div dir="ltr" id="yui_3_16_0_ym19_1_1519085807369_70649"><br></div><div dir="ltr" id="yui_3_16_0_ym19_1_1519085807369_70648">VDSM.log 2</div><div dir="ltr" id="yui_3_16_0_ym19_1_1519085807369_70650">2018-02-20 14:41:14,598+0800 INFO (periodic/3) [vdsm.api] FINISH repoStats return={'f3e372e3-1251-4195-a4b9-1027e40059df': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000252423', 'lastCheck': '6.0', 'valid': True}, 'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '7.5567e-05', 'lastCheck': '6.0', 'valid': True}, '65ca2e2d-b472-4bee-85b4-09a161464b20': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '8.2433e-05', 'lastCheck': '6.0', 'valid': True}, '4bf2ba2f-f57a-4d9f-b42a-fb78f440a358': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000237247', 'lastCheck': '5.9', 'valid': True}, '42e591b7-f86c-4b67-a3d2-40cc007f7662': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '8.4376e-05', 'lastCheck': '6.0', 'valid': True}} from=internal, task_id=6213712b-9903-4db8-9836-3baf85cd63e4 (api:52)<br id="yui_3_16_0_ym19_1_1519085807369_70683">2018-02-20 14:41:18,074+0800 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539)<br id="yui_3_16_0_ym19_1_1519085807369_70684">2018-02-20 14:41:24,333+0800 INFO (jsonrpc/4) [vdsm.api] START repoStats(options=None) from=::ffff:10.10.43.1,56540, flow_id=73f86113, task_id=e8231ecb-3543-4d8f-af54-4cf2b06ee98a (api:46)<br id="yui_3_16_0_ym19_1_1519085807369_70685">2018-02-20 14:41:24,334+0800 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'f3e372e3-1251-4195-a4b9-1027e40059df': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000172953', 'lastCheck': '5.7', 'valid': True}, 'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '7.701e-05', 'lastCheck': '5.7', 'valid': True}, '65ca2e2d-b472-4bee-85b4-09a161464b20': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000117677', 'lastCheck': '5.7', 'valid': True}, '4bf2ba2f-f57a-4d9f-b42a-fb78f440a358': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000185146', 'lastCheck': '5.7', 'valid': True}, '42e591b7-f86c-4b67-a3d2-40cc007f7662': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '9.1467e-05', 'lastCheck': '5.8', 'valid': True}} from=::ffff:10.10.43.1,56540, flow_id=73f86113, task_id=e8231ecb-3543-4d8f-af54-4cf2b06ee98a (api:52)<br id="yui_3_16_0_ym19_1_1519085807369_70686">2018-02-20 14:41:24,338+0800 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.00 seconds (__init__:539)<br id="yui_3_16_0_ym19_1_1519085807369_70687">2018-02-20 14:41:27,087+0800 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539)<br id="yui_3_16_0_ym19_1_1519085807369_70688">2018-02-20 14:41:29,607+0800 INFO (periodic/1) [vdsm.api] START repoStats(options=None) from=internal, task_id=f14b7aab-b64a-4903-9368-d665e39b49d1 (api:46)<br id="yui_3_16_0_ym19_1_1519085807369_70689">2018-02-20 14:41:29,608+0800 INFO (periodic/1) [vdsm.api] FINISH repoStats return={'f3e372e3-1251-4195-a4b9-1027e40059df': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000246237', 'lastCheck': '1.0', 'valid': True}, 'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '8.8773e-05', 'lastCheck': '1.0', 'valid': True}, '65ca2e2d-b472-4bee-85b4-09a161464b20': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '9.7145e-05', 'lastCheck': '1.0', 'valid': True}, '4bf2ba2f-f57a-4d9f-b42a-fb78f440a358': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000218729', 'lastCheck': '0.9', 'valid': True}, '42e591b7-f86c-4b67-a3d2-40cc007f7662': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '9.1035e-05', 'lastCheck': '1.0', 'valid': True}} from=internal, task_id=f14b7aab-b64a-4903-9368-d665e39b49d1 (api:52)<br id="yui_3_16_0_ym19_1_1519085807369_70690">2018-02-20 14:41:33,079+0800 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539)<br id="yui_3_16_0_ym19_1_1519085807369_70691">2018-02-20 14:41:40,456+0800 INFO (jsonrpc/2) [vdsm.api] START repoStats(options=None) from=::ffff:10.10.43.1,56540, flow_id=3be8150c, task_id=1b0c86a8-6fd8-4882-a742-fbd56ccb4037 (api:46)<br id="yui_3_16_0_ym19_1_1519085807369_70692">2018-02-20 14:41:40,457+0800 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'f3e372e3-1251-4195-a4b9-1027e40059df': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000174027', 'lastCheck': '1.8', 'valid': True}, 'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00011454', 'lastCheck': '1.8', 'valid': True}, '65ca2e2d-b472-4bee-85b4-09a161464b20': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000145955', 'lastCheck': '1.8', 'valid': True}, '4bf2ba2f-f57a-4d9f-b42a-fb78f440a358': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000209871', 'lastCheck': '1.8', 'valid': True}, '42e591b7-f86c-4b67-a3d2-40cc007f7662': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '9.5744e-05', 'lastCheck': '1.9', 'valid': True}} from=::ffff:10.10.43.1,56540, flow_id=3be8150c, task_id=1b0c86a8-6fd8-4882-a742-fbd56ccb4037 (api:52)<br id="yui_3_16_0_ym19_1_1519085807369_70693">2018-02-20 14:41:40,461+0800 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:539)<br id="yui_3_16_0_ym19_1_1519085807369_70694">2018-02-20 14:41:42,106+0800 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539)<br id="yui_3_16_0_ym19_1_1519085807369_70695">2018-02-20 14:41:44,622+0800 INFO (periodic/3) [vdsm.api] START repoStats(options=None) from=internal, task_id=20e0b29b-d3cd-4e44-b92c-213f9c984ab2 (api:46)<br id="yui_3_16_0_ym19_1_1519085807369_70696">2018-02-20 14:41:44,622+0800 INFO (periodic/3) [vdsm.api] FINISH repoStats return={'f3e372e3-1251-4195-a4b9-1027e40059df': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000174027', 'lastCheck': '6.0', 'valid': True}, 'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00011454', 'lastCheck': '6.0', 'valid': True}, '65ca2e2d-b472-4bee-85b4-09a161464b20': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000145955', 'lastCheck': '6.0', 'valid': True}, '4bf2ba2f-f57a-4d9f-b42a-fb78f440a358': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000209871', 'lastCheck': '5.9', 'valid': True}, '42e591b7-f86c-4b67-a3d2-40cc007f7662': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '9.5744e-05', 'lastCheck': '6.0', 'valid': True}} from=internal, task_id=20e0b29b-d3cd-4e44-b92c-213f9c984ab2 (api:52)<br id="yui_3_16_0_ym19_1_1519085807369_70697">2018-02-20 14:41:49,083+0800 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539)<br id="yui_3_16_0_ym19_1_1519085807369_70698"></div><div dir="ltr"><br></div><div dir="ltr" id="yui_3_16_0_ym19_1_1519085807369_70767"><br></div><div dir="ltr" id="yui_3_16_0_ym19_1_1519085807369_70768">Engine log</div><div dir="ltr"><br></div><div dir="ltr" id="yui_3_16_0_ym19_1_1519085807369_70769">skId '404ccecc-aa7f-45ea-89e4-726956269bc9' task status 'finished'<br id="yui_3_16_0_ym19_1_1519085807369_70808">2018-02-20 14:42:21,966+08 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (org.ovirt.thread.pool-7-thread-38) [29528f9] spmStart polling ended, spm status: SPM<br id="yui_3_16_0_ym19_1_1519085807369_70809">2018-02-20 14:42:21,967+08 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-38) [29528f9] START, HSMClearTaskVDSCommand(HostName = Node1, HSMTaskGuidBaseVDSCommandParameters:{runAsync='true', hostId='7dee35bb-8c97-4f6a-b6cd-abc4258540e4', taskId='404ccecc-aa7f-45ea-89e4-726956269bc9'}), log id: 71688f70<br id="yui_3_16_0_ym19_1_1519085807369_70810">2018-02-20 14:42:22,922+08 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-38) [29528f9] FINISH, HSMClearTaskVDSCommand, log id: 71688f70<br id="yui_3_16_0_ym19_1_1519085807369_70811">2018-02-20 14:42:22,923+08 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (org.ovirt.thread.pool-7-thread-38) [29528f9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@78332453, log id: 3ea35d5<br id="yui_3_16_0_ym19_1_1519085807369_70812">2018-02-20 14:42:22,935+08 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-7-thread-38) [29528f9] Initialize Irs proxy from vds: dev2node1.lares.com.ph<br id="yui_3_16_0_ym19_1_1519085807369_70813">2018-02-20 14:42:22,951+08 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-38) [29528f9] EVENT_ID: IRS_HOSTED_ON_VDS(204), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host Node1 (Address: dev2node1.lares.com.ph).<br id="yui_3_16_0_ym19_1_1519085807369_70814">2018-02-20 14:42:22,952+08 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-38) [29528f9] -- executeIrsBrokerCommand: Attempting on storage pool '5a865884-0366-0330-02b8-0000000002d4'<br id="yui_3_16_0_ym19_1_1519085807369_70815">2018-02-20 14:42:22,952+08 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-38) [29528f9] START, HSMGetAllTasksInfoVDSCommand(HostName = Node1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='7dee35bb-8c97-4f6a-b6cd-abc4258540e4'}), log id: 1bdbea9d<br id="yui_3_16_0_ym19_1_1519085807369_70816">2018-02-20 14:42:22,955+08 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-7) [29528f9] START, SPMGetAllTasksInfoVDSCommand( IrsBaseVDSCommandParameters:{runAsync='true', storagePoolId='5a865884-0366-0330-02b8-0000000002d4', ignoreFailoverLimit='false'}), log id: 5c2422d6<br id="yui_3_16_0_ym19_1_1519085807369_70817">2018-02-20 14:42:23,956+08 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-38) [29528f9] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1bdbea9d<br id="yui_3_16_0_ym19_1_1519085807369_70818">2018-02-20 14:42:23,956+08 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-38) [29528f9] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 58cbe7b7<br id="yui_3_16_0_ym19_1_1519085807369_70819">2018-02-20 14:42:23,956+08 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-7-thread-38) [29528f9] Discovered no tasks on Storage Pool 'UnsecuredEnv'<br id="yui_3_16_0_ym19_1_1519085807369_70820">2018-02-20 14:42:24,936+08 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to dev2node1.lares.com.ph/10.10.43.2<br id="yui_3_16_0_ym19_1_1519085807369_70821">2018-02-20 14:42:27,012+08 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-7-thread-43) [] Master domain is not in sync between DB and VDSM. Domain Node1Container marked as master in DB and not in the storage<br id="yui_3_16_0_ym19_1_1519085807369_70822">2018-02-20 14:42:27,026+08 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-43) [] EVENT_ID: SYSTEM_MASTER_DOMAIN_NOT_IN_SYNC(990), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Sync Error on Master Domain between Host Node1 and oVirt Engine. Domain: Node1Container is marked as Master in oVirt Engine database but not on the Storage side. Please consult with Support on how to fix this issue.<br id="yui_3_16_0_ym19_1_1519085807369_70823">2018-02-20 14:42:27,103+08 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-43) [3e5965ca] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: f3e372e3-1251-4195-a4b9-1027e40059df Type: Storage<br id="yui_3_16_0_ym19_1_1519085807369_70824">2018-02-20 14:42:27,137+08 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-7-thread-43) [3e5965ca] START, ResetIrsVDSCommand( ResetIrsVDSCommandParameters:{runAsync='true', storagePoolId='5a865884-0366-0330-02b8-0000000002d4', ignoreFailoverLimit='false', vdsId='7dee35bb-8c97-4f6a-b6cd-abc4258540e4', ignoreStopFailed='true'}), log id: 3e0a239d<br id="yui_3_16_0_ym19_1_1519085807369_70825">2018-02-20 14:42:27,140+08 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-7-thread-43) [3e5965ca] START, SpmStopVDSCommand(HostName = Node1, SpmStopVDSCommandParameters:{runAsync='true', hostId='7dee35bb-8c97-4f6a-b6cd-abc4258540e4', storagePoolId='5a865884-0366-0330-02b8-0000000002d4'}), log id: 7c67bf06<br id="yui_3_16_0_ym19_1_1519085807369_70826">2018-02-20 14:42:28,144+08 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-7-thread-43) [3e5965ca] SpmStopVDSCommand::Stopping SPM on vds 'Node1', pool id '5a865884-0366-0330-02b8-0000000002d4'<br></div><div id="yui_3_16_0_ym19_1_1519085807369_70529"><span></span></div> <div class="qtdSeparateBR"><br><br></div><div class="yahoo_quoted" style="display: block;"> <div style="font-family: Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 10px;"> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 16px;"> <div dir="ltr"><font size="2" face="Arial"> On Tuesday, February 20, 2018 2:33 PM, michael pagdanganan <mhke_aj5566@yahoo.com> wrote:<br></font></div> <br><br> <div class="y_msg_container"><div id="yiv0115319588"><div><div style="color:#000;background-color:#fff;font-family:Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:10px;"><div id="yiv0115319588yui_3_16_0_ym19_1_1519085807369_60672"><font size="3">Thanks for quick response,</font></div><div id="yiv0115319588yui_3_16_0_ym19_1_1519085807369_62803"><font size="3"><br clear="none"></font></div><div id="yiv0115319588yui_3_16_0_ym19_1_1519085807369_64423"><font size="3">see attachment.<br clear="none"></font></div><div id="yiv0115319588yui_3_16_0_ym19_1_1519085807369_60552"><span></span></div> <div class="yiv0115319588qtdSeparateBR"><br clear="none"><br clear="none"></div><div class="yiv0115319588yahoo_quoted" style="display:block;"> <div style="font-family:Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:10px;"> <div style="font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px;"> <div class="yiv0115319588yqt2980525560" id="yiv0115319588yqtfd88886"><div dir="ltr"><font size="2" face="Arial"> On Tuesday, February 20, 2018 2:10 PM, Eyal Shenitzky <eshenitz@redhat.com> wrote:<br clear="none"></font></div> <br clear="none"><br clear="none"> <div class="yiv0115319588y_msg_container"><div id="yiv0115319588"><div><div dir="ltr">Hi, <div><br clear="none"></div><div>Can you please attach full Engine and VDSM logs?</div><div><br clear="none"></div><div>Thanks,</div></div><div class="yiv0115319588gmail_extra"><br clear="none"><div class="yiv0115319588gmail_quote">On Tue, Feb 20, 2018 at 2:59 AM, michael pagdanganan <span dir="ltr"><<a rel="nofollow" shape="rect" ymailto="mailto:mhke_aj5566@yahoo.com" target="_blank" href="mailto:mhke_aj5566@yahoo.com">mhke_aj5566@yahoo.com</a>></span> wrote:<br clear="none"><blockquote class="yiv0115319588gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="yiv0115319588yqt7135196070" id="yiv0115319588yqt29383"><div><div style="color:#000;background-color:#fff;font-family:Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:10px;"><div dir="ltr" id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10542"><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_15967" size="3">My storage pool has 2 Master Domain(</font><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_16063" size="3"><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_15968" size="3">Stored2 on Node2,Node1Container on Node1)</font>. and my old master domain </font><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_15934" size="3"><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_16064" size="3">(</font><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_16065" size="3"><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_16066" size="3">DATANd01 on Node1 )</font></font> hung on preparing for maintenance. When I tried to activate my old master domain (</font><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3"><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_15935" size="3">DATANd01 on Node1 )</font> all storage domain goes down and up master keep on rotating.</font></div><div id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10905"><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3"><br clear="none"></font></div><div id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10692"><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3"><br clear="none"></font></div><div dir="ltr" id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10710"><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3">Ovirt Version (oVirt Engine Version: 4.1.9.1.el7.centos)<br clear="none"></font></div><div dir="ltr" id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_14202"><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3"><br clear="none"></font></div><div dir="ltr" id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_12604"><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3"><br clear="none"></font></div><div dir="ltr" id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_14203"><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3">Event Error:</font></div><div dir="ltr" id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_14215"><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3">Sync Error on Master Domain between Host Node2 and oVirt Engine. Domain Stored2 is marked as master in ovirt engine Database but not on storage side Please consult with support<br id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_12607" clear="none"></font></div><div dir="ltr" id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_14204"><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3"><br clear="none"></font></div><div dir="ltr" id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_15891"><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3">VDSM Node2 command ConnectStoragePoolVDS failed: Wrong Master domain or it's version: u'SD=f3e372e3-1251-4195-a4b9- 1027e40059df, pool=5a865884-0366-0330-02b8- 0000</font></div><div dir="ltr" id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_15895"><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3"><br clear="none"></font></div><div dir="ltr" id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_15896"><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3">VDSM Node2 command HSMGetAllTastsStatusesVDS failed: Not SPM: ()<br id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_15901" clear="none"><br id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_15902" clear="none">Failed to deactivate Storage Domain DATANd01 (Data Center UnsecuredEnv)<br clear="none"></font></div><div id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10545"><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3"><br clear="none"></font></div><div id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10551"><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3">Here's logs from engine:</font></div><div id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10620"><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3"><br clear="none"></font></div><div id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10573"><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3">------------------------------ ------------------------------ ------------------------------ -------------</font></div><div dir="ltr" id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10588"><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3">[root@dev2engine ~]# tail /var/log/messages<br id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10609" clear="none">Feb 20 07:01:01 dev2engine systemd: Starting Session 20 of user root.<br id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10610" clear="none">Feb 20 07:01:01 dev2engine systemd: Removed slice User Slice of root.<br id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10611" clear="none">Feb 20 07:01:01 dev2engine systemd: Stopping User Slice of root.<br id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10612" clear="none">Feb 20 07:58:52 dev2engine systemd: Created slice User Slice of root.<br id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10613" clear="none">Feb 20 07:58:52 dev2engine systemd: Starting User Slice of root.<br id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10614" clear="none">Feb 20 07:58:52 dev2engine systemd-logind: New session 21 of user root.<br id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10615" clear="none">Feb 20 07:58:52 dev2engine systemd: Started Session 21 of user root.<br id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10616" clear="none">Feb 20 07:58:52 dev2engine systemd: Starting Session 21 of user root.<br id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10617" clear="none">Feb 20 08:01:01 dev2engine systemd: Started Session 22 of user root.<br id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10618" clear="none">Feb 20 08:01:01 dev2engine systemd: Starting Session 22 of user root.</font></div><div dir="ltr" id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10619"><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3"><br clear="none"></font></div><div dir="ltr" id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10623"><br clear="none"><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3"><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10626" size="3">------------------------------ ------------------------------ ------------------------------ -------------</font></font></div><div dir="ltr" id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10654"><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3"><font id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10626" size="3">[root@dev2engine ~]# tail /var/log/ovirt-engine/engine. log<br id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10677" clear="none">2018-02-20 08:01:16,062+08 INFO [org.ovirt.engine.core.bll. eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7- thread-32) [102e9d3c] Finished reconstruct for pool '5a865884-0366-0330-02b8- 0000000002d4'. Clearing event queue<br id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10678" clear="none">2018-02-20 08:01:27,825+08 WARN [org.ovirt.engine.core. vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-7- thread-23) [] Master domain is not in sync between DB and VDSM. Domain Stored2 marked as master in DB and not in the storage<br id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10679" clear="none">2018-02-20 08:01:27,862+08 WARN [org.ovirt.engine.core.bll. storage.pool. ReconstructMasterDomainCommand ] (org.ovirt.thread.pool-7- thread-23) [213f42b9] Validation of action 'ReconstructMasterDomain' failed for user SYSTEM. Reasons: VAR__ACTION__RECONSTRUCT_ MASTER,VAR__TYPE__STORAGE__ DOMAIN,ACTION_TYPE_FAILED_ STORAGE_DOMAIN_STATUS_ ILLEGAL2,$status PreparingForMaintenance<br id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10680" clear="none">2018-02-20 08:01:27,882+08 INFO [org.ovirt.engine.core.bll. eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7- thread-20) [929330e] Finished reconstruct for pool '5a865884-0366-0330-02b8- 0000000002d4'. Clearing event queue<br id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10681" clear="none">2018-02-20 08:01:40,106+08 WARN [org.ovirt.engine.core. vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-7- thread-17) [] Master domain is not in sync between DB and VDSM. Domain Stored2 marked as master in DB and not in the storage<br id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10682" clear="none">2018-02-20 08:01:40,197+08 WARN [org.ovirt.engine.core.bll. storage.pool. ReconstructMasterDomainCommand ] (org.ovirt.thread.pool-7- thread-17) [7af552c1] Validation of action 'ReconstructMasterDomain' failed for user SYSTEM. Reasons: VAR__ACTION__RECONSTRUCT_ MASTER,VAR__TYPE__STORAGE__ DOMAIN,ACTION_TYPE_FAILED_ STORAGE_DOMAIN_STATUS_ ILLEGAL2,$status PreparingForMaintenance<br id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10683" clear="none">2018-02-20 08:01:40,246+08 INFO [org.ovirt.engine.core.bll. eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7- thread-22) [73673040] Finished reconstruct for pool '5a865884-0366-0330-02b8- 0000000002d4'. Clearing event queue<br id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10684" clear="none">2018-02-20 08:01:51,809+08 WARN [org.ovirt.engine.core. vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-7- thread-26) [] Master domain is not in sync between DB and VDSM. Domain Stored2 marked as master in DB and not in the storage<br id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10685" clear="none">2018-02-20 08:01:51,846+08 WARN [org.ovirt.engine.core.bll. storage.pool. ReconstructMasterDomainCommand ] (org.ovirt.thread.pool-7- thread-26) [20307cbe] Validation of action 'ReconstructMasterDomain' failed for user SYSTEM. Reasons: VAR__ACTION__RECONSTRUCT_ MASTER,VAR__TYPE__STORAGE__ DOMAIN,ACTION_TYPE_FAILED_ STORAGE_DOMAIN_STATUS_ ILLEGAL2,$status PreparingForMaintenance<br id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10686" clear="none">2018-02-20 08:01:51,866+08 INFO [org.ovirt.engine.core.bll. eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7- thread-49) [2c11a866] Finished reconstruct for pool '5a865884-0366-0330-02b8- 0000000002d4'. Clearing event queue<br id="yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10687" clear="none"><br clear="none"></font></font></div></div></div></div><br clear="none">______________________________ _________________<br clear="none">
Users mailing list<br clear="none">
<a rel="nofollow" shape="rect" ymailto="mailto:Users@ovirt.org" target="_blank" href="mailto:Users@ovirt.org">Users@ovirt.org</a><br clear="none">
<a rel="nofollow" shape="rect" target="_blank" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/ mailman/listinfo/users</a><br clear="none">
<br clear="none"></blockquote></div><br clear="none"><br clear="all"><div><br clear="none"></div>-- <br clear="none"><div class="yiv0115319588gmail_signature"><div dir="ltr">Regards,<div>Eyal Shenitzky</div></div></div>
</div></div></div><br clear="none"><br clear="none"></div> </div></div><div class="yiv0115319588yqt2980525560" id="yiv0115319588yqtfd94549"> </div></div><div class="yiv0115319588yqt2980525560" id="yiv0115319588yqtfd67072"> </div></div></div></div></div><br><br></div> </div> </div> </div></div></body></html>