[ovirt-users] 2 Master on Storage Pool [Event Error]

Tue Feb 20 06:43:07 UTC 2018

Sorry can't attached log file it's too big file

VDSM.log for node 1
2766', 'lastCheck': '4.9', 'valid': True}} from=internal, task_id=645d456e-f59f-4b1c-9e97-fc82d19a36b1 (api:52)
2018-02-20 14:38:47,222+0800 INFO  (jsonrpc/3) [vdsm.api] START repoStats(options=None) from=::ffff:10.10.43.1,60554, flow_id=3b2e802e, task_id=28c795d1-1639-4e68-a9fe-a00006be268f (api:46)
2018-02-20 14:38:47,222+0800 INFO  (jsonrpc/3) [vdsm.api] FINISH repoStats return={'f3e372e3-1251-4195-a4b9-1027e40059df': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000119109', 'lastCheck': '6.0', 'valid': True}, 'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000195263', 'lastCheck': '6.0', 'valid': True}, '65ca2e2d-b472-4bee-85b4-09a161464b20': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00022988', 'lastCheck': '6.0', 'valid': True}, '4bf2ba2f-f57a-4d9f-b42a-fb78f440a358': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000270206', 'lastCheck': '5.9', 'valid': True}, '42e591b7-f86c-4b67-a3d2-40cc007f7662': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000272766', 'lastCheck': '6.0', 'valid': True}} from=::ffff:10.10.43.1,60554, flow_id=3b2e802e, task_id=28c795d1-1639-4e68-a9fe-a00006be268f (api:52)
2018-02-20 14:38:47,226+0800 INFO  (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.00 seconds (__init__:539)
2018-02-20 14:38:55,252+0800 INFO  (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539)
2018-02-20 14:38:58,566+0800 INFO  (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539)
2018-02-20 14:39:01,093+0800 INFO  (periodic/1) [vdsm.api] START repoStats(options=None) from=internal, task_id=04f55ded-5841-44ae-a376-4f6e723e4b10 (api:46)
2018-02-20 14:39:01,093+0800 INFO  (periodic/1) [vdsm.api] FINISH repoStats return={'f3e372e3-1251-4195-a4b9-1027e40059df': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '9.3888e-05', 'lastCheck': '9.9', 'valid': True}, 'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000197535', 'lastCheck': '9.8', 'valid': True}, '65ca2e2d-b472-4bee-85b4-09a161464b20': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000191456', 'lastCheck': '9.8', 'valid': True}, '4bf2ba2f-f57a-4d9f-b42a-fb78f440a358': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00026365', 'lastCheck': '9.8', 'valid': True}, '42e591b7-f86c-4b67-a3d2-40cc007f7662': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000243658', 'lastCheck': '0.0', 'valid': True}} from=internal, task_id=04f55ded-5841-44ae-a376-4f6e723e4b10 (api:52)
2018-02-20 14:39:02,295+0800 INFO  (jsonrpc/6) [vdsm.api] START repoStats(options=None) from=::ffff:10.10.43.1,60554, flow_id=13149d6e, task_id=5f19d76a-d343-4dc4-ad09-024ec27f7443 (api:46)
2018-02-20 14:39:02,295+0800 INFO  (jsonrpc/6) [vdsm.api] FINISH repoStats return={'f3e372e3-1251-4195-a4b9-1027e40059df': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '9.4759e-05', 'lastCheck': '1.1', 'valid': True}, 'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000183158', 'lastCheck': '1.0', 'valid': True}, '65ca2e2d-b472-4bee-85b4-09a161464b20': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000222609', 'lastCheck': '1.0', 'valid': True}, '4bf2ba2f-f57a-4d9f-b42a-fb78f440a358': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000211253', 'lastCheck': '1.0', 'valid': True}, '42e591b7-f86c-4b67-a3d2-40cc007f7662': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000243658', 'lastCheck': '1.1', 'valid': True}} from=::ffff:10.10.43.1,60554, flow_id=13149d6e, task_id=5f19d76a-d343-4dc4-ad09-024ec27f7443 (api:52)
2018-02-20 14:39:02,300+0800 INFO  (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:539)
2018-02-20 14:39:10,270+0800 INFO  (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539)
2018-02-20 14:39:13,631+0800 INFO  (jsonrpc/0) [vdsm.api] START connectStorageServer(domType=1, spUUID=u'00000000-0000-0000-0000-000000000000', conList=[{u'id': u'5e4c94f1-f3b9-4fbd-a6c7-e732d0fe3123', u'connection': u'dev2node1.lares.com.ph:/run/media/root/Slave1Data/dataNode1', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'protocol_version': u'auto', u'password': '********', u'port': u''}], options=None) from=::ffff:10.10.43.1,60554, flow_id=15b57417, task_id=bfce1e70-ef4a-4e13-aaaa-7a66aaf44429 (api:46)
2018-02-20 14:39:13,633+0800 INFO  (jsonrpc/0) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'status': 0, 'id': u'5e4c94f1-f3b9-4fbd-a6c7-e732d0fe3123'}]} from=::ffff:10.10.43.1,60554, flow_id=15b57417, task_id=bfce1e70-ef4a-4e13-aaaa-7a66aaf44429 (api:52)
2018-02-20 14:39:13,634+0800 INFO  (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer succeeded in 0.00 seconds (__init__:539)
2018-02-20 14:39:13,806+0800 INFO  (jsonrpc/7) [vdsm.api] START connectStoragePool(spUUID=u'5a865884-0366-0330-02b8-0000000002d4', hostID=1, msdUUID=u'f3e372e3-1251-4195-a4b9-1027e40059df', masterVersion=65, domainsMap={u'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': u'active', u'4bf2ba2f-f57a-4d9f-b42a-fb78f440a358': u'active', u'225e1975-8121-4370-b317-86e964ae326f': u'attached', u'f3e372e3-1251-4195-a4b9-1027e40059df': u'active', u'65ca2e2d-b472-4bee-85b4-09a161464b20': u'active', u'42e591b7-f86c-4b67-a3d2-40cc007f7662': u'active'}, options=None) from=::ffff:10.10.43.1,60554, flow_id=15b57417, task_id=927a5d9a-4304-4776-b5c9-22ba5d0cb853 (api:46)
2018-02-20 14:39:13,807+0800 INFO  (jsonrpc/7) [storage.StoragePoolMemoryBackend] new storage pool master version 65 and domains map {u'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': u'Active', u'4bf2ba2f-f57a-4d9f-b42a-fb78f440a358': u'Active', u'225e1975-8121-4370-b317-86e964ae326f': u'Attached', u'f3e372e3-1251-4195-a4b9-1027e40059df': u'Active', u'65ca2e2d-b472-4bee-85b4-09a161464b20': u'Active', u'42e591b7-f86c-4b67-a3d2-40cc007f7662': u'Active'} (spbackends:450)

VDSM.log 22018-02-20 14:41:14,598+0800 INFO  (periodic/3) [vdsm.api] FINISH repoStats return={'f3e372e3-1251-4195-a4b9-1027e40059df': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000252423', 'lastCheck': '6.0', 'valid': True}, 'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '7.5567e-05', 'lastCheck': '6.0', 'valid': True}, '65ca2e2d-b472-4bee-85b4-09a161464b20': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '8.2433e-05', 'lastCheck': '6.0', 'valid': True}, '4bf2ba2f-f57a-4d9f-b42a-fb78f440a358': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000237247', 'lastCheck': '5.9', 'valid': True}, '42e591b7-f86c-4b67-a3d2-40cc007f7662': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '8.4376e-05', 'lastCheck': '6.0', 'valid': True}} from=internal, task_id=6213712b-9903-4db8-9836-3baf85cd63e4 (api:52)
2018-02-20 14:41:18,074+0800 INFO  (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539)
2018-02-20 14:41:24,333+0800 INFO  (jsonrpc/4) [vdsm.api] START repoStats(options=None) from=::ffff:10.10.43.1,56540, flow_id=73f86113, task_id=e8231ecb-3543-4d8f-af54-4cf2b06ee98a (api:46)
2018-02-20 14:41:24,334+0800 INFO  (jsonrpc/4) [vdsm.api] FINISH repoStats return={'f3e372e3-1251-4195-a4b9-1027e40059df': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000172953', 'lastCheck': '5.7', 'valid': True}, 'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '7.701e-05', 'lastCheck': '5.7', 'valid': True}, '65ca2e2d-b472-4bee-85b4-09a161464b20': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000117677', 'lastCheck': '5.7', 'valid': True}, '4bf2ba2f-f57a-4d9f-b42a-fb78f440a358': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000185146', 'lastCheck': '5.7', 'valid': True}, '42e591b7-f86c-4b67-a3d2-40cc007f7662': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '9.1467e-05', 'lastCheck': '5.8', 'valid': True}} from=::ffff:10.10.43.1,56540, flow_id=73f86113, task_id=e8231ecb-3543-4d8f-af54-4cf2b06ee98a (api:52)
2018-02-20 14:41:24,338+0800 INFO  (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.00 seconds (__init__:539)
2018-02-20 14:41:27,087+0800 INFO  (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539)
2018-02-20 14:41:29,607+0800 INFO  (periodic/1) [vdsm.api] START repoStats(options=None) from=internal, task_id=f14b7aab-b64a-4903-9368-d665e39b49d1 (api:46)
2018-02-20 14:41:29,608+0800 INFO  (periodic/1) [vdsm.api] FINISH repoStats return={'f3e372e3-1251-4195-a4b9-1027e40059df': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000246237', 'lastCheck': '1.0', 'valid': True}, 'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '8.8773e-05', 'lastCheck': '1.0', 'valid': True}, '65ca2e2d-b472-4bee-85b4-09a161464b20': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '9.7145e-05', 'lastCheck': '1.0', 'valid': True}, '4bf2ba2f-f57a-4d9f-b42a-fb78f440a358': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000218729', 'lastCheck': '0.9', 'valid': True}, '42e591b7-f86c-4b67-a3d2-40cc007f7662': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '9.1035e-05', 'lastCheck': '1.0', 'valid': True}} from=internal, task_id=f14b7aab-b64a-4903-9368-d665e39b49d1 (api:52)
2018-02-20 14:41:33,079+0800 INFO  (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539)
2018-02-20 14:41:40,456+0800 INFO  (jsonrpc/2) [vdsm.api] START repoStats(options=None) from=::ffff:10.10.43.1,56540, flow_id=3be8150c, task_id=1b0c86a8-6fd8-4882-a742-fbd56ccb4037 (api:46)
2018-02-20 14:41:40,457+0800 INFO  (jsonrpc/2) [vdsm.api] FINISH repoStats return={'f3e372e3-1251-4195-a4b9-1027e40059df': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000174027', 'lastCheck': '1.8', 'valid': True}, 'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00011454', 'lastCheck': '1.8', 'valid': True}, '65ca2e2d-b472-4bee-85b4-09a161464b20': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000145955', 'lastCheck': '1.8', 'valid': True}, '4bf2ba2f-f57a-4d9f-b42a-fb78f440a358': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000209871', 'lastCheck': '1.8', 'valid': True}, '42e591b7-f86c-4b67-a3d2-40cc007f7662': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '9.5744e-05', 'lastCheck': '1.9', 'valid': True}} from=::ffff:10.10.43.1,56540, flow_id=3be8150c, task_id=1b0c86a8-6fd8-4882-a742-fbd56ccb4037 (api:52)
2018-02-20 14:41:40,461+0800 INFO  (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:539)
2018-02-20 14:41:42,106+0800 INFO  (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539)
2018-02-20 14:41:44,622+0800 INFO  (periodic/3) [vdsm.api] START repoStats(options=None) from=internal, task_id=20e0b29b-d3cd-4e44-b92c-213f9c984ab2 (api:46)
2018-02-20 14:41:44,622+0800 INFO  (periodic/3) [vdsm.api] FINISH repoStats return={'f3e372e3-1251-4195-a4b9-1027e40059df': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000174027', 'lastCheck': '6.0', 'valid': True}, 'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00011454', 'lastCheck': '6.0', 'valid': True}, '65ca2e2d-b472-4bee-85b4-09a161464b20': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000145955', 'lastCheck': '6.0', 'valid': True}, '4bf2ba2f-f57a-4d9f-b42a-fb78f440a358': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000209871', 'lastCheck': '5.9', 'valid': True}, '42e591b7-f86c-4b67-a3d2-40cc007f7662': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '9.5744e-05', 'lastCheck': '6.0', 'valid': True}} from=internal, task_id=20e0b29b-d3cd-4e44-b92c-213f9c984ab2 (api:52)
2018-02-20 14:41:49,083+0800 INFO  (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539)

Engine log
skId '404ccecc-aa7f-45ea-89e4-726956269bc9' task status 'finished'
2018-02-20 14:42:21,966+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (org.ovirt.thread.pool-7-thread-38) [29528f9] spmStart polling ended, spm status: SPM
2018-02-20 14:42:21,967+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-38) [29528f9] START, HSMClearTaskVDSCommand(HostName = Node1, HSMTaskGuidBaseVDSCommandParameters:{runAsync='true', hostId='7dee35bb-8c97-4f6a-b6cd-abc4258540e4', taskId='404ccecc-aa7f-45ea-89e4-726956269bc9'}), log id: 71688f70
2018-02-20 14:42:22,922+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-38) [29528f9] FINISH, HSMClearTaskVDSCommand, log id: 71688f70
2018-02-20 14:42:22,923+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (org.ovirt.thread.pool-7-thread-38) [29528f9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult at 78332453, log id: 3ea35d5
2018-02-20 14:42:22,935+08 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-7-thread-38) [29528f9] Initialize Irs proxy from vds: dev2node1.lares.com.ph
2018-02-20 14:42:22,951+08 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-38) [29528f9] EVENT_ID: IRS_HOSTED_ON_VDS(204), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host Node1 (Address: dev2node1.lares.com.ph).
2018-02-20 14:42:22,952+08 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-38) [29528f9] -- executeIrsBrokerCommand: Attempting on storage pool '5a865884-0366-0330-02b8-0000000002d4'
2018-02-20 14:42:22,952+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-38) [29528f9] START, HSMGetAllTasksInfoVDSCommand(HostName = Node1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='7dee35bb-8c97-4f6a-b6cd-abc4258540e4'}), log id: 1bdbea9d
2018-02-20 14:42:22,955+08 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-7) [29528f9] START, SPMGetAllTasksInfoVDSCommand( IrsBaseVDSCommandParameters:{runAsync='true', storagePoolId='5a865884-0366-0330-02b8-0000000002d4', ignoreFailoverLimit='false'}), log id: 5c2422d6
2018-02-20 14:42:23,956+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-38) [29528f9] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1bdbea9d
2018-02-20 14:42:23,956+08 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-38) [29528f9] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 58cbe7b7
2018-02-20 14:42:23,956+08 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-7-thread-38) [29528f9] Discovered no tasks on Storage Pool 'UnsecuredEnv'
2018-02-20 14:42:24,936+08 INFO  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to dev2node1.lares.com.ph/10.10.43.2
2018-02-20 14:42:27,012+08 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-7-thread-43) [] Master domain is not in sync between DB and VDSM. Domain Node1Container marked as master in DB and not in the storage
2018-02-20 14:42:27,026+08 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-43) [] EVENT_ID: SYSTEM_MASTER_DOMAIN_NOT_IN_SYNC(990), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Sync Error on Master Domain between Host Node1 and oVirt Engine. Domain: Node1Container is marked as Master in oVirt Engine database but not on the Storage side. Please consult with Support on how to fix this issue.
2018-02-20 14:42:27,103+08 INFO  [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-43) [3e5965ca] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: f3e372e3-1251-4195-a4b9-1027e40059df Type: Storage
2018-02-20 14:42:27,137+08 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-7-thread-43) [3e5965ca] START, ResetIrsVDSCommand( ResetIrsVDSCommandParameters:{runAsync='true', storagePoolId='5a865884-0366-0330-02b8-0000000002d4', ignoreFailoverLimit='false', vdsId='7dee35bb-8c97-4f6a-b6cd-abc4258540e4', ignoreStopFailed='true'}), log id: 3e0a239d
2018-02-20 14:42:27,140+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-7-thread-43) [3e5965ca] START, SpmStopVDSCommand(HostName = Node1, SpmStopVDSCommandParameters:{runAsync='true', hostId='7dee35bb-8c97-4f6a-b6cd-abc4258540e4', storagePoolId='5a865884-0366-0330-02b8-0000000002d4'}), log id: 7c67bf06
2018-02-20 14:42:28,144+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-7-thread-43) [3e5965ca] SpmStopVDSCommand::Stopping SPM on vds 'Node1', pool id '5a865884-0366-0330-02b8-0000000002d4'

    On Tuesday, February 20, 2018 2:33 PM, michael pagdanganan <mhke_aj5566 at yahoo.com> wrote:

 Thanks for quick response,
see attachment.

    On Tuesday, February 20, 2018 2:10 PM, Eyal Shenitzky <eshenitz at redhat.com> wrote:

 Hi, 
Can you please attach full Engine and VDSM logs?
Thanks,
On Tue, Feb 20, 2018 at 2:59 AM, michael pagdanganan <mhke_aj5566 at yahoo.com> wrote:

My storage pool has 2 Master Domain(Stored2 on Node2,Node1Container on Node1). and my old master domain (DATANd01 on Node1 ) hung on preparing for maintenance. When I tried to activate my old master domain (DATANd01 on Node1 ) all  storage domain goes down and up master keep on rotating.

Ovirt Version (oVirt Engine Version: 4.1.9.1.el7.centos)

Event Error:Sync Error on Master Domain between Host Node2 and oVirt Engine. Domain Stored2 is marked as master in ovirt engine Database but not on storage side Please consult with support

VDSM Node2 command ConnectStoragePoolVDS failed: Wrong Master domain or it's version: u'SD=f3e372e3-1251-4195-a4b9- 1027e40059df, pool=5a865884-0366-0330-02b8- 0000
VDSM Node2 command HSMGetAllTastsStatusesVDS failed: Not SPM: ()

Failed to deactivate Storage Domain DATANd01 (Data Center UnsecuredEnv)

Here's logs from engine:
------------------------------ ------------------------------ ------------------------------ -------------[root at dev2engine ~]# tail /var/log/messages
Feb 20 07:01:01 dev2engine systemd: Starting Session 20 of user root.
Feb 20 07:01:01 dev2engine systemd: Removed slice User Slice of root.
Feb 20 07:01:01 dev2engine systemd: Stopping User Slice of root.
Feb 20 07:58:52 dev2engine systemd: Created slice User Slice of root.
Feb 20 07:58:52 dev2engine systemd: Starting User Slice of root.
Feb 20 07:58:52 dev2engine systemd-logind: New session 21 of user root.
Feb 20 07:58:52 dev2engine systemd: Started Session 21 of user root.
Feb 20 07:58:52 dev2engine systemd: Starting Session 21 of user root.
Feb 20 08:01:01 dev2engine systemd: Started Session 22 of user root.
Feb 20 08:01:01 dev2engine systemd: Starting Session 22 of user root.

------------------------------ ------------------------------ ------------------------------ -------------[root at dev2engine ~]# tail /var/log/ovirt-engine/engine. log
2018-02-20 08:01:16,062+08 INFO  [org.ovirt.engine.core.bll. eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7- thread-32) [102e9d3c] Finished reconstruct for pool '5a865884-0366-0330-02b8- 0000000002d4'. Clearing event queue
2018-02-20 08:01:27,825+08 WARN  [org.ovirt.engine.core. vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-7- thread-23) [] Master domain is not in sync between DB and VDSM. Domain Stored2 marked as master in DB and not in the storage
2018-02-20 08:01:27,862+08 WARN  [org.ovirt.engine.core.bll. storage.pool. ReconstructMasterDomainCommand ] (org.ovirt.thread.pool-7- thread-23) [213f42b9] Validation of action 'ReconstructMasterDomain' failed for user SYSTEM. Reasons: VAR__ACTION__RECONSTRUCT_ MASTER,VAR__TYPE__STORAGE__ DOMAIN,ACTION_TYPE_FAILED_ STORAGE_DOMAIN_STATUS_ ILLEGAL2,$status PreparingForMaintenance
2018-02-20 08:01:27,882+08 INFO  [org.ovirt.engine.core.bll. eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7- thread-20) [929330e] Finished reconstruct for pool '5a865884-0366-0330-02b8- 0000000002d4'. Clearing event queue
2018-02-20 08:01:40,106+08 WARN  [org.ovirt.engine.core. vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-7- thread-17) [] Master domain is not in sync between DB and VDSM. Domain Stored2 marked as master in DB and not in the storage
2018-02-20 08:01:40,197+08 WARN  [org.ovirt.engine.core.bll. storage.pool. ReconstructMasterDomainCommand ] (org.ovirt.thread.pool-7- thread-17) [7af552c1] Validation of action 'ReconstructMasterDomain' failed for user SYSTEM. Reasons: VAR__ACTION__RECONSTRUCT_ MASTER,VAR__TYPE__STORAGE__ DOMAIN,ACTION_TYPE_FAILED_ STORAGE_DOMAIN_STATUS_ ILLEGAL2,$status PreparingForMaintenance
2018-02-20 08:01:40,246+08 INFO  [org.ovirt.engine.core.bll. eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7- thread-22) [73673040] Finished reconstruct for pool '5a865884-0366-0330-02b8- 0000000002d4'. Clearing event queue
2018-02-20 08:01:51,809+08 WARN  [org.ovirt.engine.core. vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-7- thread-26) [] Master domain is not in sync between DB and VDSM. Domain Stored2 marked as master in DB and not in the storage
2018-02-20 08:01:51,846+08 WARN  [org.ovirt.engine.core.bll. storage.pool. ReconstructMasterDomainCommand ] (org.ovirt.thread.pool-7- thread-26) [20307cbe] Validation of action 'ReconstructMasterDomain' failed for user SYSTEM. Reasons: VAR__ACTION__RECONSTRUCT_ MASTER,VAR__TYPE__STORAGE__ DOMAIN,ACTION_TYPE_FAILED_ STORAGE_DOMAIN_STATUS_ ILLEGAL2,$status PreparingForMaintenance
2018-02-20 08:01:51,866+08 INFO  [org.ovirt.engine.core.bll. eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7- thread-49) [2c11a866] Finished reconstruct for pool '5a865884-0366-0330-02b8- 0000000002d4'. Clearing event queue

______________________________ _________________
Users mailing list
Users at ovirt.org
http://lists.ovirt.org/ mailman/listinfo/users

-- 
Regards,Eyal Shenitzky

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20180220/321c44d0/attachment.html>