Re: [ovirt-users] 2 Master on Storage Pool [Event Error]

------=_Part_2010355_281080504.1519108987392 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Sorry can't attached log file it's too big file VDSM.log for node 1 2766', 'lastCheck': '4.9', 'valid': True}} from=3Dinternal, task_id=3D645d4= 56e-f59f-4b1c-9e97-fc82d19a36b1 (api:52) 2018-02-20 14:38:47,222+0800 INFO=C2=A0 (jsonrpc/3) [vdsm.api] START repoSt= ats(options=3DNone) from=3D::ffff:10.10.43.1,60554, flow_id=3D3b2e802e, tas= k_id=3D28c795d1-1639-4e68-a9fe-a00006be268f (api:46) 2018-02-20 14:38:47,222+0800 INFO=C2=A0 (jsonrpc/3) [vdsm.api] FINISH repoS= tats return=3D{'f3e372e3-1251-4195-a4b9-1027e40059df': {'code': 0, 'actual'= : True, 'version': 4, 'acquired': True, 'delay': '0.000119109', 'lastCheck'= : '6.0', 'valid': True}, 'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': {'code': 0= , 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000195263', '= lastCheck': '6.0', 'valid': True}, '65ca2e2d-b472-4bee-85b4-09a161464b20': = {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000= 22988', 'lastCheck': '6.0', 'valid': True}, '4bf2ba2f-f57a-4d9f-b42a-fb78f4= 40a358': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay= ': '0.000270206', 'lastCheck': '5.9', 'valid': True}, '42e591b7-f86c-4b67-a= 3d2-40cc007f7662': {'code': 0, 'actual': True, 'version': 4, 'acquired': Tr= ue, 'delay': '0.000272766', 'lastCheck': '6.0', 'valid': True}} from=3D::ff= ff:10.10.43.1,60554, flow_id=3D3b2e802e, task_id=3D28c795d1-1639-4e68-a9fe-= a00006be268f (api:52) 2018-02-20 14:38:47,226+0800 INFO=C2=A0 (jsonrpc/3) [jsonrpc.JsonRpcServer]= RPC call Host.getStats succeeded in 0.00 seconds (__init__:539) 2018-02-20 14:38:55,252+0800 INFO=C2=A0 (jsonrpc/4) [jsonrpc.JsonRpcServer]= RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2018-02-20 14:38:58,566+0800 INFO=C2=A0 (jsonrpc/2) [jsonrpc.JsonRpcServer]= RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2018-02-20 14:39:01,093+0800 INFO=C2=A0 (periodic/1) [vdsm.api] START repoS= tats(options=3DNone) from=3Dinternal, task_id=3D04f55ded-5841-44ae-a376-4f6= e723e4b10 (api:46) 2018-02-20 14:39:01,093+0800 INFO=C2=A0 (periodic/1) [vdsm.api] FINISH repo= Stats return=3D{'f3e372e3-1251-4195-a4b9-1027e40059df': {'code': 0, 'actual= ': True, 'version': 4, 'acquired': True, 'delay': '9.3888e-05', 'lastCheck'= : '9.9', 'valid': True}, 'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': {'code': 0= , 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000197535', '= lastCheck': '9.8', 'valid': True}, '65ca2e2d-b472-4bee-85b4-09a161464b20': = {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000= 191456', 'lastCheck': '9.8', 'valid': True}, '4bf2ba2f-f57a-4d9f-b42a-fb78f= 440a358': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'dela= y': '0.00026365', 'lastCheck': '9.8', 'valid': True}, '42e591b7-f86c-4b67-a= 3d2-40cc007f7662': {'code': 0, 'actual': True, 'version': 4, 'acquired': Tr= ue, 'delay': '0.000243658', 'lastCheck': '0.0', 'valid': True}} from=3Dinte= rnal, task_id=3D04f55ded-5841-44ae-a376-4f6e723e4b10 (api:52) 2018-02-20 14:39:02,295+0800 INFO=C2=A0 (jsonrpc/6) [vdsm.api] START repoSt= ats(options=3DNone) from=3D::ffff:10.10.43.1,60554, flow_id=3D13149d6e, tas= k_id=3D5f19d76a-d343-4dc4-ad09-024ec27f7443 (api:46) 2018-02-20 14:39:02,295+0800 INFO=C2=A0 (jsonrpc/6) [vdsm.api] FINISH repoS= tats return=3D{'f3e372e3-1251-4195-a4b9-1027e40059df': {'code': 0, 'actual'= : True, 'version': 4, 'acquired': True, 'delay': '9.4759e-05', 'lastCheck':= '1.1', 'valid': True}, 'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': {'code': 0,= 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000183158', 'l= astCheck': '1.0', 'valid': True}, '65ca2e2d-b472-4bee-85b4-09a161464b20': {= 'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.0002= 22609', 'lastCheck': '1.0', 'valid': True}, '4bf2ba2f-f57a-4d9f-b42a-fb78f4= 40a358': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay= ': '0.000211253', 'lastCheck': '1.0', 'valid': True}, '42e591b7-f86c-4b67-a= 3d2-40cc007f7662': {'code': 0, 'actual': True, 'version': 4, 'acquired': Tr= ue, 'delay': '0.000243658', 'lastCheck': '1.1', 'valid': True}} from=3D::ff= ff:10.10.43.1,60554, flow_id=3D13149d6e, task_id=3D5f19d76a-d343-4dc4-ad09-= 024ec27f7443 (api:52) 2018-02-20 14:39:02,300+0800 INFO=C2=A0 (jsonrpc/6) [jsonrpc.JsonRpcServer]= RPC call Host.getStats succeeded in 0.01 seconds (__init__:539) 2018-02-20 14:39:10,270+0800 INFO=C2=A0 (jsonrpc/1) [jsonrpc.JsonRpcServer]= RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2018-02-20 14:39:13,631+0800 INFO=C2=A0 (jsonrpc/0) [vdsm.api] START connec= tStorageServer(domType=3D1, spUUID=3Du'00000000-0000-0000-0000-000000000000= ', conList=3D[{u'id': u'5e4c94f1-f3b9-4fbd-a6c7-e732d0fe3123', u'connection= ': u'dev2node1.lares.com.ph:/run/media/root/Slave1Data/dataNode1', u'iqn': = u'', u'user': u'', u'tpgt': u'1', u'protocol_version': u'auto', u'password'= : '********', u'port': u''}], options=3DNone) from=3D::ffff:10.10.43.1,6055= 4, flow_id=3D15b57417, task_id=3Dbfce1e70-ef4a-4e13-aaaa-7a66aaf44429 (api:= 46) 2018-02-20 14:39:13,633+0800 INFO=C2=A0 (jsonrpc/0) [vdsm.api] FINISH conne= ctStorageServer return=3D{'statuslist': [{'status': 0, 'id': u'5e4c94f1-f3b= 9-4fbd-a6c7-e732d0fe3123'}]} from=3D::ffff:10.10.43.1,60554, flow_id=3D15b5= 7417, task_id=3Dbfce1e70-ef4a-4e13-aaaa-7a66aaf44429 (api:52) 2018-02-20 14:39:13,634+0800 INFO=C2=A0 (jsonrpc/0) [jsonrpc.JsonRpcServer]= RPC call StoragePool.connectStorageServer succeeded in 0.00 seconds (__ini= t__:539) 2018-02-20 14:39:13,806+0800 INFO=C2=A0 (jsonrpc/7) [vdsm.api] START connec= tStoragePool(spUUID=3Du'5a865884-0366-0330-02b8-0000000002d4', hostID=3D1, = msdUUID=3Du'f3e372e3-1251-4195-a4b9-1027e40059df', masterVersion=3D65, doma= insMap=3D{u'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': u'active', u'4bf2ba2f-f5= 7a-4d9f-b42a-fb78f440a358': u'active', u'225e1975-8121-4370-b317-86e964ae32= 6f': u'attached', u'f3e372e3-1251-4195-a4b9-1027e40059df': u'active', u'65c= a2e2d-b472-4bee-85b4-09a161464b20': u'active', u'42e591b7-f86c-4b67-a3d2-40= cc007f7662': u'active'}, options=3DNone) from=3D::ffff:10.10.43.1,60554, fl= ow_id=3D15b57417, task_id=3D927a5d9a-4304-4776-b5c9-22ba5d0cb853 (api:46) 2018-02-20 14:39:13,807+0800 INFO=C2=A0 (jsonrpc/7) [storage.StoragePoolMem= oryBackend] new storage pool master version 65 and domains map {u'e83d0d46-= 6ea6-4aa3-80bf-6e95c66b0454': u'Active', u'4bf2ba2f-f57a-4d9f-b42a-fb78f440= a358': u'Active', u'225e1975-8121-4370-b317-86e964ae326f': u'Attached', u'f= 3e372e3-1251-4195-a4b9-1027e40059df': u'Active', u'65ca2e2d-b472-4bee-85b4-= 09a161464b20': u'Active', u'42e591b7-f86c-4b67-a3d2-40cc007f7662': u'Active= '} (spbackends:450) VDSM.log 22018-02-20 14:41:14,598+0800 INFO=C2=A0 (periodic/3) [vdsm.api] F= INISH repoStats return=3D{'f3e372e3-1251-4195-a4b9-1027e40059df': {'code': = 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000252423', = 'lastCheck': '6.0', 'valid': True}, 'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454':= {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '7.55= 67e-05', 'lastCheck': '6.0', 'valid': True}, '65ca2e2d-b472-4bee-85b4-09a16= 1464b20': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'dela= y': '8.2433e-05', 'lastCheck': '6.0', 'valid': True}, '4bf2ba2f-f57a-4d9f-b= 42a-fb78f440a358': {'code': 0, 'actual': True, 'version': 0, 'acquired': Tr= ue, 'delay': '0.000237247', 'lastCheck': '5.9', 'valid': True}, '42e591b7-f= 86c-4b67-a3d2-40cc007f7662': {'code': 0, 'actual': True, 'version': 4, 'acq= uired': True, 'delay': '8.4376e-05', 'lastCheck': '6.0', 'valid': True}} fr= om=3Dinternal, task_id=3D6213712b-9903-4db8-9836-3baf85cd63e4 (api:52) 2018-02-20 14:41:18,074+0800 INFO=C2=A0 (jsonrpc/6) [jsonrpc.JsonRpcServer]= RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2018-02-20 14:41:24,333+0800 INFO=C2=A0 (jsonrpc/4) [vdsm.api] START repoSt= ats(options=3DNone) from=3D::ffff:10.10.43.1,56540, flow_id=3D73f86113, tas= k_id=3De8231ecb-3543-4d8f-af54-4cf2b06ee98a (api:46) 2018-02-20 14:41:24,334+0800 INFO=C2=A0 (jsonrpc/4) [vdsm.api] FINISH repoS= tats return=3D{'f3e372e3-1251-4195-a4b9-1027e40059df': {'code': 0, 'actual'= : True, 'version': 4, 'acquired': True, 'delay': '0.000172953', 'lastCheck'= : '5.7', 'valid': True}, 'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': {'code': 0= , 'actual': True, 'version': 0, 'acquired': True, 'delay': '7.701e-05', 'la= stCheck': '5.7', 'valid': True}, '65ca2e2d-b472-4bee-85b4-09a161464b20': {'= code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00011= 7677', 'lastCheck': '5.7', 'valid': True}, '4bf2ba2f-f57a-4d9f-b42a-fb78f44= 0a358': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay'= : '0.000185146', 'lastCheck': '5.7', 'valid': True}, '42e591b7-f86c-4b67-a3= d2-40cc007f7662': {'code': 0, 'actual': True, 'version': 4, 'acquired': Tru= e, 'delay': '9.1467e-05', 'lastCheck': '5.8', 'valid': True}} from=3D::ffff= :10.10.43.1,56540, flow_id=3D73f86113, task_id=3De8231ecb-3543-4d8f-af54-4c= f2b06ee98a (api:52) 2018-02-20 14:41:24,338+0800 INFO=C2=A0 (jsonrpc/4) [jsonrpc.JsonRpcServer]= RPC call Host.getStats succeeded in 0.00 seconds (__init__:539) 2018-02-20 14:41:27,087+0800 INFO=C2=A0 (jsonrpc/0) [jsonrpc.JsonRpcServer]= RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2018-02-20 14:41:29,607+0800 INFO=C2=A0 (periodic/1) [vdsm.api] START repoS= tats(options=3DNone) from=3Dinternal, task_id=3Df14b7aab-b64a-4903-9368-d66= 5e39b49d1 (api:46) 2018-02-20 14:41:29,608+0800 INFO=C2=A0 (periodic/1) [vdsm.api] FINISH repo= Stats return=3D{'f3e372e3-1251-4195-a4b9-1027e40059df': {'code': 0, 'actual= ': True, 'version': 4, 'acquired': True, 'delay': '0.000246237', 'lastCheck= ': '1.0', 'valid': True}, 'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': {'code': = 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '8.8773e-05', '= lastCheck': '1.0', 'valid': True}, '65ca2e2d-b472-4bee-85b4-09a161464b20': = {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '9.714= 5e-05', 'lastCheck': '1.0', 'valid': True}, '4bf2ba2f-f57a-4d9f-b42a-fb78f4= 40a358': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay= ': '0.000218729', 'lastCheck': '0.9', 'valid': True}, '42e591b7-f86c-4b67-a= 3d2-40cc007f7662': {'code': 0, 'actual': True, 'version': 4, 'acquired': Tr= ue, 'delay': '9.1035e-05', 'lastCheck': '1.0', 'valid': True}} from=3Dinter= nal, task_id=3Df14b7aab-b64a-4903-9368-d665e39b49d1 (api:52) 2018-02-20 14:41:33,079+0800 INFO=C2=A0 (jsonrpc/7) [jsonrpc.JsonRpcServer]= RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2018-02-20 14:41:40,456+0800 INFO=C2=A0 (jsonrpc/2) [vdsm.api] START repoSt= ats(options=3DNone) from=3D::ffff:10.10.43.1,56540, flow_id=3D3be8150c, tas= k_id=3D1b0c86a8-6fd8-4882-a742-fbd56ccb4037 (api:46) 2018-02-20 14:41:40,457+0800 INFO=C2=A0 (jsonrpc/2) [vdsm.api] FINISH repoS= tats return=3D{'f3e372e3-1251-4195-a4b9-1027e40059df': {'code': 0, 'actual'= : True, 'version': 4, 'acquired': True, 'delay': '0.000174027', 'lastCheck'= : '1.8', 'valid': True}, 'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': {'code': 0= , 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00011454', 'l= astCheck': '1.8', 'valid': True}, '65ca2e2d-b472-4bee-85b4-09a161464b20': {= 'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.0001= 45955', 'lastCheck': '1.8', 'valid': True}, '4bf2ba2f-f57a-4d9f-b42a-fb78f4= 40a358': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay= ': '0.000209871', 'lastCheck': '1.8', 'valid': True}, '42e591b7-f86c-4b67-a= 3d2-40cc007f7662': {'code': 0, 'actual': True, 'version': 4, 'acquired': Tr= ue, 'delay': '9.5744e-05', 'lastCheck': '1.9', 'valid': True}} from=3D::fff= f:10.10.43.1,56540, flow_id=3D3be8150c, task_id=3D1b0c86a8-6fd8-4882-a742-f= bd56ccb4037 (api:52) 2018-02-20 14:41:40,461+0800 INFO=C2=A0 (jsonrpc/2) [jsonrpc.JsonRpcServer]= RPC call Host.getStats succeeded in 0.01 seconds (__init__:539) 2018-02-20 14:41:42,106+0800 INFO=C2=A0 (jsonrpc/3) [jsonrpc.JsonRpcServer]= RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) 2018-02-20 14:41:44,622+0800 INFO=C2=A0 (periodic/3) [vdsm.api] START repoS= tats(options=3DNone) from=3Dinternal, task_id=3D20e0b29b-d3cd-4e44-b92c-213= f9c984ab2 (api:46) 2018-02-20 14:41:44,622+0800 INFO=C2=A0 (periodic/3) [vdsm.api] FINISH repo= Stats return=3D{'f3e372e3-1251-4195-a4b9-1027e40059df': {'code': 0, 'actual= ': True, 'version': 4, 'acquired': True, 'delay': '0.000174027', 'lastCheck= ': '6.0', 'valid': True}, 'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': {'code': = 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00011454', '= lastCheck': '6.0', 'valid': True}, '65ca2e2d-b472-4bee-85b4-09a161464b20': = {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000= 145955', 'lastCheck': '6.0', 'valid': True}, '4bf2ba2f-f57a-4d9f-b42a-fb78f= 440a358': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'dela= y': '0.000209871', 'lastCheck': '5.9', 'valid': True}, '42e591b7-f86c-4b67-= a3d2-40cc007f7662': {'code': 0, 'actual': True, 'version': 4, 'acquired': T= rue, 'delay': '9.5744e-05', 'lastCheck': '6.0', 'valid': True}} from=3Dinte= rnal, task_id=3D20e0b29b-d3cd-4e44-b92c-213f9c984ab2 (api:52) 2018-02-20 14:41:49,083+0800 INFO=C2=A0 (jsonrpc/1) [jsonrpc.JsonRpcServer]= RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539) Engine log skId '404ccecc-aa7f-45ea-89e4-726956269bc9' task status 'finished' 2018-02-20 14:42:21,966+08 INFO=C2=A0 [org.ovirt.engine.core.vdsbroker.vdsb= roker.SpmStartVDSCommand] (org.ovirt.thread.pool-7-thread-38) [29528f9] spm= Start polling ended, spm status: SPM 2018-02-20 14:42:21,967+08 INFO=C2=A0 [org.ovirt.engine.core.vdsbroker.vdsb= roker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-38) [29528f9]= START, HSMClearTaskVDSCommand(HostName =3D Node1, HSMTaskGuidBaseVDSComman= dParameters:{runAsync=3D'true', hostId=3D'7dee35bb-8c97-4f6a-b6cd-abc425854= 0e4', taskId=3D'404ccecc-aa7f-45ea-89e4-726956269bc9'}), log id: 71688f70 2018-02-20 14:42:22,922+08 INFO=C2=A0 [org.ovirt.engine.core.vdsbroker.vdsb= roker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-38) [29528f9]= FINISH, HSMClearTaskVDSCommand, log id: 71688f70 2018-02-20 14:42:22,923+08 INFO=C2=A0 [org.ovirt.engine.core.vdsbroker.vdsb= roker.SpmStartVDSCommand] (org.ovirt.thread.pool-7-thread-38) [29528f9] FIN= ISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentit= ies.SpmStatusResult@78332453, log id: 3ea35d5 2018-02-20 14:42:22,935+08 INFO=C2=A0 [org.ovirt.engine.core.vdsbroker.irsb= roker.IrsProxy] (org.ovirt.thread.pool-7-thread-38) [29528f9] Initialize Ir= s proxy from vds: dev2node1.lares.com.ph 2018-02-20 14:42:22,951+08 INFO=C2=A0 [org.ovirt.engine.core.dal.dbbroker.a= uditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-38) [2952= 8f9] EVENT_ID: IRS_HOSTED_ON_VDS(204), Correlation ID: null, Call Stack: nu= ll, Custom ID: null, Custom Event ID: -1, Message: Storage Pool Manager run= s on Host Node1 (Address: dev2node1.lares.com.ph). 2018-02-20 14:42:22,952+08 INFO=C2=A0 [org.ovirt.engine.core.vdsbroker.irsb= roker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-38) [29= 528f9] -- executeIrsBrokerCommand: Attempting on storage pool '5a865884-036= 6-0330-02b8-0000000002d4' 2018-02-20 14:42:22,952+08 INFO=C2=A0 [org.ovirt.engine.core.vdsbroker.vdsb= roker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-38) [29= 528f9] START, HSMGetAllTasksInfoVDSCommand(HostName =3D Node1, VdsIdVDSComm= andParametersBase:{runAsync=3D'true', hostId=3D'7dee35bb-8c97-4f6a-b6cd-abc= 4258540e4'}), log id: 1bdbea9d 2018-02-20 14:42:22,955+08 INFO=C2=A0 [org.ovirt.engine.core.vdsbroker.irsb= roker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-7) [295= 28f9] START, SPMGetAllTasksInfoVDSCommand( IrsBaseVDSCommandParameters:{run= Async=3D'true', storagePoolId=3D'5a865884-0366-0330-02b8-0000000002d4', ign= oreFailoverLimit=3D'false'}), log id: 5c2422d6 2018-02-20 14:42:23,956+08 INFO=C2=A0 [org.ovirt.engine.core.vdsbroker.vdsb= roker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-38) [29= 528f9] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1bdbea9d 2018-02-20 14:42:23,956+08 INFO=C2=A0 [org.ovirt.engine.core.vdsbroker.irsb= roker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-38) [29= 528f9] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 58cbe7b7 2018-02-20 14:42:23,956+08 INFO=C2=A0 [org.ovirt.engine.core.bll.tasks.Asyn= cTaskManager] (org.ovirt.thread.pool-7-thread-38) [29528f9] Discovered no t= asks on Storage Pool 'UnsecuredEnv' 2018-02-20 14:42:24,936+08 INFO=C2=A0 [org.ovirt.vdsm.jsonrpc.client.reacto= rs.ReactorClient] (SSL Stomp Reactor) [] Connecting to dev2node1.lares.com.= ph/10.10.43.2 2018-02-20 14:42:27,012+08 WARN=C2=A0 [org.ovirt.engine.core.vdsbroker.irsb= roker.IrsProxy] (org.ovirt.thread.pool-7-thread-43) [] Master domain is not= in sync between DB and VDSM. Domain Node1Container marked as master in DB = and not in the storage 2018-02-20 14:42:27,026+08 WARN=C2=A0 [org.ovirt.engine.core.dal.dbbroker.a= uditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-43) [] EV= ENT_ID: SYSTEM_MASTER_DOMAIN_NOT_IN_SYNC(990), Correlation ID: null, Call S= tack: null, Custom ID: null, Custom Event ID: -1, Message: Sync Error on Ma= ster Domain between Host Node1 and oVirt Engine. Domain: Node1Container is = marked as Master in oVirt Engine database but not on the Storage side. Plea= se consult with Support on how to fix this issue. 2018-02-20 14:42:27,103+08 INFO=C2=A0 [org.ovirt.engine.core.bll.storage.po= ol.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-43) [3e5= 965ca] Running command: ReconstructMasterDomainCommand internal: true. Enti= ties affected :=C2=A0 ID: f3e372e3-1251-4195-a4b9-1027e40059df Type: Storag= e 2018-02-20 14:42:27,137+08 INFO=C2=A0 [org.ovirt.engine.core.vdsbroker.irsb= roker.ResetIrsVDSCommand] (org.ovirt.thread.pool-7-thread-43) [3e5965ca] ST= ART, ResetIrsVDSCommand( ResetIrsVDSCommandParameters:{runAsync=3D'true', s= toragePoolId=3D'5a865884-0366-0330-02b8-0000000002d4', ignoreFailoverLimit= =3D'false', vdsId=3D'7dee35bb-8c97-4f6a-b6cd-abc4258540e4', ignoreStopFaile= d=3D'true'}), log id: 3e0a239d 2018-02-20 14:42:27,140+08 INFO=C2=A0 [org.ovirt.engine.core.vdsbroker.vdsb= roker.SpmStopVDSCommand] (org.ovirt.thread.pool-7-thread-43) [3e5965ca] STA= RT, SpmStopVDSCommand(HostName =3D Node1, SpmStopVDSCommandParameters:{runA= sync=3D'true', hostId=3D'7dee35bb-8c97-4f6a-b6cd-abc4258540e4', storagePool= Id=3D'5a865884-0366-0330-02b8-0000000002d4'}), log id: 7c67bf06 2018-02-20 14:42:28,144+08 INFO=C2=A0 [org.ovirt.engine.core.vdsbroker.vdsb= roker.SpmStopVDSCommand] (org.ovirt.thread.pool-7-thread-43) [3e5965ca] Spm= StopVDSCommand::Stopping SPM on vds 'Node1', pool id '5a865884-0366-0330-02= b8-0000000002d4' =20 On Tuesday, February 20, 2018 2:33 PM, michael pagdanganan <mhke_aj5566= @yahoo.com=20 Thanks for quick response, see attachment. =20 On Tuesday, February 20, 2018 2:10 PM, Eyal Shenitzky <eshenitz(a)redhat.= com=20 Hi,=C2=A0 Can you please attach full Engine and VDSM logs? Thanks, On Tue, Feb 20, 2018 at 2:59 AM, michael pagdanganan <mhke_aj5566(a)yahoo.com= My storage pool has 2 Master Domain(Stored2 on Node2,Node1Container on Node= 1). and my old master domain (DATANd01 on Node1 ) hung on preparing for mai= ntenance. When I tried to activate my old master domain (DATANd01 on Node1 = ) all=C2=A0 storage domain goes down and up master keep on rotating. Ovirt Version (oVirt Engine Version: 4.1.9.1.el7.centos) Event Error:Sync Error on Master Domain between Host Node2 and oVirt Engine= . Domain Stored2 is marked as master in ovirt engine Database but not on st= orage side Please consult with support VDSM Node2 command ConnectStoragePoolVDS failed: Wrong Master domain or it'= s version: u'SD=3Df3e372e3-1251-4195-a4b9- 1027e40059df, pool=3D5a865884-03= 66-0330-02b8- 0000 VDSM Node2 command HSMGetAllTastsStatusesVDS failed: Not SPM: () Failed to deactivate Storage Domain DATANd01 (Data Center UnsecuredEnv) Here's logs from engine: ------------------------------ ------------------------------ -------------= ----------------- -------------[root@dev2engine ~]# tail /var/log/messages Feb 20 07:01:01 dev2engine systemd: Starting Session 20 of user root. Feb 20 07:01:01 dev2engine systemd: Removed slice User Slice of root. Feb 20 07:01:01 dev2engine systemd: Stopping User Slice of root. Feb 20 07:58:52 dev2engine systemd: Created slice User Slice of root. Feb 20 07:58:52 dev2engine systemd: Starting User Slice of root. Feb 20 07:58:52 dev2engine systemd-logind: New session 21 of user root. Feb 20 07:58:52 dev2engine systemd: Started Session 21 of user root. Feb 20 07:58:52 dev2engine systemd: Starting Session 21 of user root. Feb 20 08:01:01 dev2engine systemd: Started Session 22 of user root. Feb 20 08:01:01 dev2engine systemd: Starting Session 22 of user root. ------------------------------ ------------------------------ -------------= ----------------- -------------[root@dev2engine ~]# tail /var/log/ovirt-eng= ine/engine. log 2018-02-20 08:01:16,062+08 INFO=C2=A0 [org.ovirt.engine.core.bll. eventqueu= e.EventQueueMonitor] (org.ovirt.thread.pool-7- thread-32) [102e9d3c] Finish= ed reconstruct for pool '5a865884-0366-0330-02b8- 0000000002d4'. Clearing e= vent queue 2018-02-20 08:01:27,825+08 WARN=C2=A0 [org.ovirt.engine.core. vdsbroker.irs= broker.IrsProxy] (org.ovirt.thread.pool-7- thread-23) [] Master domain is n= ot in sync between DB and VDSM. Domain Stored2 marked as master in DB and n= ot in the storage 2018-02-20 08:01:27,862+08 WARN=C2=A0 [org.ovirt.engine.core.bll. storage.p= ool. ReconstructMasterDomainCommand ] (org.ovirt.thread.pool-7- thread-23) = [213f42b9] Validation of action 'ReconstructMasterDomain' failed for user S= YSTEM. Reasons: VAR__ACTION__RECONSTRUCT_ MASTER,VAR__TYPE__STORAGE__ DOMAI= N,ACTION_TYPE_FAILED_ STORAGE_DOMAIN_STATUS_ ILLEGAL2,$status PreparingForM= aintenance 2018-02-20 08:01:27,882+08 INFO=C2=A0 [org.ovirt.engine.core.bll. eventqueu= e.EventQueueMonitor] (org.ovirt.thread.pool-7- thread-20) [929330e] Finishe= d reconstruct for pool '5a865884-0366-0330-02b8- 0000000002d4'. Clearing ev= ent queue 2018-02-20 08:01:40,106+08 WARN=C2=A0 [org.ovirt.engine.core. vdsbroker.irs= broker.IrsProxy] (org.ovirt.thread.pool-7- thread-17) [] Master domain is n= ot in sync between DB and VDSM. Domain Stored2 marked as master in DB and n= ot in the storage 2018-02-20 08:01:40,197+08 WARN=C2=A0 [org.ovirt.engine.core.bll. storage.p= ool. ReconstructMasterDomainCommand ] (org.ovirt.thread.pool-7- thread-17) = [7af552c1] Validation of action 'ReconstructMasterDomain' failed for user S= YSTEM. Reasons: VAR__ACTION__RECONSTRUCT_ MASTER,VAR__TYPE__STORAGE__ DOMAI= N,ACTION_TYPE_FAILED_ STORAGE_DOMAIN_STATUS_ ILLEGAL2,$status PreparingForM= aintenance 2018-02-20 08:01:40,246+08 INFO=C2=A0 [org.ovirt.engine.core.bll. eventqueu= e.EventQueueMonitor] (org.ovirt.thread.pool-7- thread-22) [73673040] Finish= ed reconstruct for pool '5a865884-0366-0330-02b8- 0000000002d4'. Clearing e= vent queue 2018-02-20 08:01:51,809+08 WARN=C2=A0 [org.ovirt.engine.core. vdsbroker.irs= broker.IrsProxy] (org.ovirt.thread.pool-7- thread-26) [] Master domain is n= ot in sync between DB and VDSM. Domain Stored2 marked as master in DB and n= ot in the storage 2018-02-20 08:01:51,846+08 WARN=C2=A0 [org.ovirt.engine.core.bll. storage.p= ool. ReconstructMasterDomainCommand ] (org.ovirt.thread.pool-7- thread-26) = [20307cbe] Validation of action 'ReconstructMasterDomain' failed for user S= YSTEM. Reasons: VAR__ACTION__RECONSTRUCT_ MASTER,VAR__TYPE__STORAGE__ DOMAI= N,ACTION_TYPE_FAILED_ STORAGE_DOMAIN_STATUS_ ILLEGAL2,$status PreparingForM= aintenance 2018-02-20 08:01:51,866+08 INFO=C2=A0 [org.ovirt.engine.core.bll. eventqueu= e.EventQueueMonitor] (org.ovirt.thread.pool-7- thread-49) [2c11a866] Finish= ed reconstruct for pool '5a865884-0366-0330-02b8- 0000000002d4'. Clearing e= vent queue ______________________________ _________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/ mailman/listinfo/users --=20 Regards,Eyal Shenitzky =20 =20 ------=_Part_2010355_281080504.1519108987392 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <html><head></head><body><div style=3D"color:#000; background-color:#fff; f= ont-family:Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font= -size:10px"><div id=3D"yui_3_16_0_ym19_1_1519085807369_70843">Sorry can't a= ttached log file it's too big file<br></div><div id=3D"yui_3_16_0_ym19_1_15= 19085807369_70846"><br></div><div><br></div><div>VDSM.log for node 1</div><= div id=3D"yui_3_16_0_ym19_1_1519085807369_70844"><br></div><div id=3D"yui_3= _16_0_ym19_1_1519085807369_70530" dir=3D"ltr">2766', 'lastCheck': '4.9', 'v= alid': True}} from=3Dinternal, task_id=3D645d456e-f59f-4b1c-9e97-fc82d19a36= b1 (api:52)<br id=3D"yui_3_16_0_ym19_1_1519085807369_70565">2018-02-20 14:3= 8:47,222+0800 INFO&nbsp; (jsonrpc/3) [vdsm.api] START repoStats(options=3DN= one) from=3D::ffff:10.10.43.1,60554, flow_id=3D3b2e802e, task_id=3D28c795d1= -1639-4e68-a9fe-a00006be268f (api:46)<br id=3D"yui_3_16_0_ym19_1_1519085807= 369_70566">2018-02-20 14:38:47,222+0800 INFO&nbsp; (jsonrpc/3) [vdsm.api] F= INISH repoStats return=3D{'f3e372e3-1251-4195-a4b9-1027e40059df': {'code': = 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000119109', = 'lastCheck': '6.0', 'valid': True}, 'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454':= {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00= 0195263', 'lastCheck': '6.0', 'valid': True}, '65ca2e2d-b472-4bee-85b4-09a1= 61464b20': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'del= ay': '0.00022988', 'lastCheck': '6.0', 'valid': True}, '4bf2ba2f-f57a-4d9f-= b42a-fb78f440a358': {'code': 0, 'actual': True, 'version': 0, 'acquired': T= rue, 'delay': '0.000270206', 'lastCheck': '5.9', 'valid': True}, '42e591b7-= f86c-4b67-a3d2-40cc007f7662': {'code': 0, 'actual': True, 'version': 4, 'ac= quired': True, 'delay': '0.000272766', 'lastCheck': '6.0', 'valid': True}} = from=3D::ffff:10.10.43.1,60554, flow_id=3D3b2e802e, task_id=3D28c795d1-1639= -4e68-a9fe-a00006be268f (api:52)<br id=3D"yui_3_16_0_ym19_1_1519085807369_7= 0567">2018-02-20 14:38:47,226+0800 INFO&nbsp; (jsonrpc/3) [jsonrpc.JsonRpcS= erver] RPC call Host.getStats succeeded in 0.00 seconds (__init__:539)<br i= d=3D"yui_3_16_0_ym19_1_1519085807369_70568">2018-02-20 14:38:55,252+0800 IN= FO&nbsp; (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats su= cceeded in 0.00 seconds (__init__:539)<br id=3D"yui_3_16_0_ym19_1_151908580= 7369_70569">2018-02-20 14:38:58,566+0800 INFO&nbsp; (jsonrpc/2) [jsonrpc.Js= onRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init_= _:539)<br id=3D"yui_3_16_0_ym19_1_1519085807369_70570">2018-02-20 14:39:01,= 093+0800 INFO&nbsp; (periodic/1) [vdsm.api] START repoStats(options=3DNone)= from=3Dinternal, task_id=3D04f55ded-5841-44ae-a376-4f6e723e4b10 (api:46)<b= r id=3D"yui_3_16_0_ym19_1_1519085807369_70571">2018-02-20 14:39:01,093+0800= INFO&nbsp; (periodic/1) [vdsm.api] FINISH repoStats return=3D{'f3e372e3-12= 51-4195-a4b9-1027e40059df': {'code': 0, 'actual': True, 'version': 4, 'acqu= ired': True, 'delay': '9.3888e-05', 'lastCheck': '9.9', 'valid': True}, 'e8= 3d0d46-6ea6-4aa3-80bf-6e95c66b0454': {'code': 0, 'actual': True, 'version':= 0, 'acquired': True, 'delay': '0.000197535', 'lastCheck': '9.8', 'valid': = True}, '65ca2e2d-b472-4bee-85b4-09a161464b20': {'code': 0, 'actual': True, = 'version': 4, 'acquired': True, 'delay': '0.000191456', 'lastCheck': '9.8',= 'valid': True}, '4bf2ba2f-f57a-4d9f-b42a-fb78f440a358': {'code': 0, 'actua= l': True, 'version': 0, 'acquired': True, 'delay': '0.00026365', 'lastCheck= ': '9.8', 'valid': True}, '42e591b7-f86c-4b67-a3d2-40cc007f7662': {'code': = 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000243658', = 'lastCheck': '0.0', 'valid': True}} from=3Dinternal, task_id=3D04f55ded-584= 1-44ae-a376-4f6e723e4b10 (api:52)<br id=3D"yui_3_16_0_ym19_1_1519085807369_= 70572">2018-02-20 14:39:02,295+0800 INFO&nbsp; (jsonrpc/6) [vdsm.api] START= repoStats(options=3DNone) from=3D::ffff:10.10.43.1,60554, flow_id=3D13149d= 6e, task_id=3D5f19d76a-d343-4dc4-ad09-024ec27f7443 (api:46)<br id=3D"yui_3_= 16_0_ym19_1_1519085807369_70573">2018-02-20 14:39:02,295+0800 INFO&nbsp; (j= sonrpc/6) [vdsm.api] FINISH repoStats return=3D{'f3e372e3-1251-4195-a4b9-10= 27e40059df': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'd= elay': '9.4759e-05', 'lastCheck': '1.1', 'valid': True}, 'e83d0d46-6ea6-4aa= 3-80bf-6e95c66b0454': {'code': 0, 'actual': True, 'version': 0, 'acquired':= True, 'delay': '0.000183158', 'lastCheck': '1.0', 'valid': True}, '65ca2e2= d-b472-4bee-85b4-09a161464b20': {'code': 0, 'actual': True, 'version': 4, '= acquired': True, 'delay': '0.000222609', 'lastCheck': '1.0', 'valid': True}= , '4bf2ba2f-f57a-4d9f-b42a-fb78f440a358': {'code': 0, 'actual': True, 'vers= ion': 0, 'acquired': True, 'delay': '0.000211253', 'lastCheck': '1.0', 'val= id': True}, '42e591b7-f86c-4b67-a3d2-40cc007f7662': {'code': 0, 'actual': T= rue, 'version': 4, 'acquired': True, 'delay': '0.000243658', 'lastCheck': '= 1.1', 'valid': True}} from=3D::ffff:10.10.43.1,60554, flow_id=3D13149d6e, t= ask_id=3D5f19d76a-d343-4dc4-ad09-024ec27f7443 (api:52)<br id=3D"yui_3_16_0_= ym19_1_1519085807369_70574">2018-02-20 14:39:02,300+0800 INFO&nbsp; (jsonrp= c/6) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 secon= ds (__init__:539)<br id=3D"yui_3_16_0_ym19_1_1519085807369_70575">2018-02-2= 0 14:39:10,270+0800 INFO&nbsp; (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call= Host.getAllVmStats succeeded in 0.00 seconds (__init__:539)<br id=3D"yui_3= _16_0_ym19_1_1519085807369_70576">2018-02-20 14:39:13,631+0800 INFO&nbsp; (= jsonrpc/0) [vdsm.api] START connectStorageServer(domType=3D1, spUUID=3Du'00= 000000-0000-0000-0000-000000000000', conList=3D[{u'id': u'5e4c94f1-f3b9-4fb= d-a6c7-e732d0fe3123', u'connection': u'dev2node1.lares.com.ph:/run/media/ro= ot/Slave1Data/dataNode1', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'proto= col_version': u'auto', u'password': '********', u'port': u''}], options=3DN= one) from=3D::ffff:10.10.43.1,60554, flow_id=3D15b57417, task_id=3Dbfce1e70= -ef4a-4e13-aaaa-7a66aaf44429 (api:46)<br id=3D"yui_3_16_0_ym19_1_1519085807= 369_70577">2018-02-20 14:39:13,633+0800 INFO&nbsp; (jsonrpc/0) [vdsm.api] F= INISH connectStorageServer return=3D{'statuslist': [{'status': 0, 'id': u'5= e4c94f1-f3b9-4fbd-a6c7-e732d0fe3123'}]} from=3D::ffff:10.10.43.1,60554, flo= w_id=3D15b57417, task_id=3Dbfce1e70-ef4a-4e13-aaaa-7a66aaf44429 (api:52)<br= id=3D"yui_3_16_0_ym19_1_1519085807369_70578">2018-02-20 14:39:13,634+0800 = INFO&nbsp; (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.connect= StorageServer succeeded in 0.00 seconds (__init__:539)<br id=3D"yui_3_16_0_= ym19_1_1519085807369_70579">2018-02-20 14:39:13,806+0800 INFO&nbsp; (jsonrp= c/7) [vdsm.api] START connectStoragePool(spUUID=3Du'5a865884-0366-0330-02b8= -0000000002d4', hostID=3D1, msdUUID=3Du'f3e372e3-1251-4195-a4b9-1027e40059d= f', masterVersion=3D65, domainsMap=3D{u'e83d0d46-6ea6-4aa3-80bf-6e95c66b045= 4': u'active', u'4bf2ba2f-f57a-4d9f-b42a-fb78f440a358': u'active', u'225e19= 75-8121-4370-b317-86e964ae326f': u'attached', u'f3e372e3-1251-4195-a4b9-102= 7e40059df': u'active', u'65ca2e2d-b472-4bee-85b4-09a161464b20': u'active', = u'42e591b7-f86c-4b67-a3d2-40cc007f7662': u'active'}, options=3DNone) from= =3D::ffff:10.10.43.1,60554, flow_id=3D15b57417, task_id=3D927a5d9a-4304-477= 6-b5c9-22ba5d0cb853 (api:46)<br id=3D"yui_3_16_0_ym19_1_1519085807369_70580= ">2018-02-20 14:39:13,807+0800 INFO&nbsp; (jsonrpc/7) [storage.StoragePoolM= emoryBackend] new storage pool master version 65 and domains map {u'e83d0d4= 6-6ea6-4aa3-80bf-6e95c66b0454': u'Active', u'4bf2ba2f-f57a-4d9f-b42a-fb78f4= 40a358': u'Active', u'225e1975-8121-4370-b317-86e964ae326f': u'Attached', u= 'f3e372e3-1251-4195-a4b9-1027e40059df': u'Active', u'65ca2e2d-b472-4bee-85b= 4-09a161464b20': u'Active', u'42e591b7-f86c-4b67-a3d2-40cc007f7662': u'Acti= ve'} (spbackends:450)<br id=3D"yui_3_16_0_ym19_1_1519085807369_70581"></div= iv dir=3D"ltr" id=3D"yui_3_16_0_ym19_1_1519085807369_70648">VDSM.log 2</div= 4:41:14,598+0800 INFO&nbsp; (periodic/3) [vdsm.api] FINISH repoStats return= =3D{'f3e372e3-1251-4195-a4b9-1027e40059df': {'code': 0, 'actual': True, 've= rsion': 4, 'acquired': True, 'delay': '0.000252423', 'lastCheck': '6.0', 'v= alid': True}, 'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': {'code': 0, 'actual':= True, 'version': 0, 'acquired': True, 'delay': '7.5567e-05', 'lastCheck': = '6.0', 'valid': True}, '65ca2e2d-b472-4bee-85b4-09a161464b20': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '8.2433e-05', 'las= tCheck': '6.0', 'valid': True}, '4bf2ba2f-f57a-4d9f-b42a-fb78f440a358': {'c= ode': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000237= 247', 'lastCheck': '5.9', 'valid': True}, '42e591b7-f86c-4b67-a3d2-40cc007f= 7662': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay':= '8.4376e-05', 'lastCheck': '6.0', 'valid': True}} from=3Dinternal, task_id= =3D6213712b-9903-4db8-9836-3baf85cd63e4 (api:52)<br id=3D"yui_3_16_0_ym19_1= _1519085807369_70683">2018-02-20 14:41:18,074+0800 INFO&nbsp; (jsonrpc/6) [= jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 second= s (__init__:539)<br id=3D"yui_3_16_0_ym19_1_1519085807369_70684">2018-02-20= 14:41:24,333+0800 INFO&nbsp; (jsonrpc/4) [vdsm.api] START repoStats(option= s=3DNone) from=3D::ffff:10.10.43.1,56540, flow_id=3D73f86113, task_id=3De82= 31ecb-3543-4d8f-af54-4cf2b06ee98a (api:46)<br id=3D"yui_3_16_0_ym19_1_15190= 85807369_70685">2018-02-20 14:41:24,334+0800 INFO&nbsp; (jsonrpc/4) [vdsm.a= pi] FINISH repoStats return=3D{'f3e372e3-1251-4195-a4b9-1027e40059df': {'co= de': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.0001729= 53', 'lastCheck': '5.7', 'valid': True}, 'e83d0d46-6ea6-4aa3-80bf-6e95c66b0= 454': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': = '7.701e-05', 'lastCheck': '5.7', 'valid': True}, '65ca2e2d-b472-4bee-85b4-0= 9a161464b20': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, '= delay': '0.000117677', 'lastCheck': '5.7', 'valid': True}, '4bf2ba2f-f57a-4= d9f-b42a-fb78f440a358': {'code': 0, 'actual': True, 'version': 0, 'acquired= ': True, 'delay': '0.000185146', 'lastCheck': '5.7', 'valid': True}, '42e59= 1b7-f86c-4b67-a3d2-40cc007f7662': {'code': 0, 'actual': True, 'version': 4,= 'acquired': True, 'delay': '9.1467e-05', 'lastCheck': '5.8', 'valid': True= }} from=3D::ffff:10.10.43.1,56540, flow_id=3D73f86113, task_id=3De8231ecb-3= 543-4d8f-af54-4cf2b06ee98a (api:52)<br id=3D"yui_3_16_0_ym19_1_151908580736= 9_70686">2018-02-20 14:41:24,338+0800 INFO&nbsp; (jsonrpc/4) [jsonrpc.JsonR= pcServer] RPC call Host.getStats succeeded in 0.00 seconds (__init__:539)<b= r id=3D"yui_3_16_0_ym19_1_1519085807369_70687">2018-02-20 14:41:27,087+0800= INFO&nbsp; (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats= succeeded in 0.00 seconds (__init__:539)<br id=3D"yui_3_16_0_ym19_1_151908= 5807369_70688">2018-02-20 14:41:29,607+0800 INFO&nbsp; (periodic/1) [vdsm.a= pi] START repoStats(options=3DNone) from=3Dinternal, task_id=3Df14b7aab-b64= a-4903-9368-d665e39b49d1 (api:46)<br id=3D"yui_3_16_0_ym19_1_1519085807369_= 70689">2018-02-20 14:41:29,608+0800 INFO&nbsp; (periodic/1) [vdsm.api] FINI= SH repoStats return=3D{'f3e372e3-1251-4195-a4b9-1027e40059df': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000246237', 'la= stCheck': '1.0', 'valid': True}, 'e83d0d46-6ea6-4aa3-80bf-6e95c66b0454': {'= code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '8.8773e= -05', 'lastCheck': '1.0', 'valid': True}, '65ca2e2d-b472-4bee-85b4-09a16146= 4b20': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay':= '9.7145e-05', 'lastCheck': '1.0', 'valid': True}, '4bf2ba2f-f57a-4d9f-b42a= -fb78f440a358': {'code': 0, 'actual': True, 'version': 0, 'acquired': True,= 'delay': '0.000218729', 'lastCheck': '0.9', 'valid': True}, '42e591b7-f86c= -4b67-a3d2-40cc007f7662': {'code': 0, 'actual': True, 'version': 4, 'acquir= ed': True, 'delay': '9.1035e-05', 'lastCheck': '1.0', 'valid': True}} from= =3Dinternal, task_id=3Df14b7aab-b64a-4903-9368-d665e39b49d1 (api:52)<br id= =3D"yui_3_16_0_ym19_1_1519085807369_70690">2018-02-20 14:41:33,079+0800 INF= O&nbsp; (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats suc= ceeded in 0.00 seconds (__init__:539)<br id=3D"yui_3_16_0_ym19_1_1519085807= 369_70691">2018-02-20 14:41:40,456+0800 INFO&nbsp; (jsonrpc/2) [vdsm.api] S= TART repoStats(options=3DNone) from=3D::ffff:10.10.43.1,56540, flow_id=3D3b= e8150c, task_id=3D1b0c86a8-6fd8-4882-a742-fbd56ccb4037 (api:46)<br id=3D"yu= i_3_16_0_ym19_1_1519085807369_70692">2018-02-20 14:41:40,457+0800 INFO&nbsp= ; (jsonrpc/2) [vdsm.api] FINISH repoStats return=3D{'f3e372e3-1251-4195-a4b= 9-1027e40059df': {'code': 0, 'actual': True, 'version': 4, 'acquired': True= , 'delay': '0.000174027', 'lastCheck': '1.8', 'valid': True}, 'e83d0d46-6ea= 6-4aa3-80bf-6e95c66b0454': {'code': 0, 'actual': True, 'version': 0, 'acqui= red': True, 'delay': '0.00011454', 'lastCheck': '1.8', 'valid': True}, '65c= a2e2d-b472-4bee-85b4-09a161464b20': {'code': 0, 'actual': True, 'version': = 4, 'acquired': True, 'delay': '0.000145955', 'lastCheck': '1.8', 'valid': T= rue}, '4bf2ba2f-f57a-4d9f-b42a-fb78f440a358': {'code': 0, 'actual': True, '= version': 0, 'acquired': True, 'delay': '0.000209871', 'lastCheck': '1.8', = 'valid': True}, '42e591b7-f86c-4b67-a3d2-40cc007f7662': {'code': 0, 'actual= ': True, 'version': 4, 'acquired': True, 'delay': '9.5744e-05', 'lastCheck'= : '1.9', 'valid': True}} from=3D::ffff:10.10.43.1,56540, flow_id=3D3be8150c= , task_id=3D1b0c86a8-6fd8-4882-a742-fbd56ccb4037 (api:52)<br id=3D"yui_3_16= _0_ym19_1_1519085807369_70693">2018-02-20 14:41:40,461+0800 INFO&nbsp; (jso= nrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 se= conds (__init__:539)<br id=3D"yui_3_16_0_ym19_1_1519085807369_70694">2018-0= 2-20 14:41:42,106+0800 INFO&nbsp; (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC c= all Host.getAllVmStats succeeded in 0.00 seconds (__init__:539)<br id=3D"yu= i_3_16_0_ym19_1_1519085807369_70695">2018-02-20 14:41:44,622+0800 INFO&nbsp= ; (periodic/3) [vdsm.api] START repoStats(options=3DNone) from=3Dinternal, = task_id=3D20e0b29b-d3cd-4e44-b92c-213f9c984ab2 (api:46)<br id=3D"yui_3_16_0= _ym19_1_1519085807369_70696">2018-02-20 14:41:44,622+0800 INFO&nbsp; (perio= dic/3) [vdsm.api] FINISH repoStats return=3D{'f3e372e3-1251-4195-a4b9-1027e= 40059df': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'dela= y': '0.000174027', 'lastCheck': '6.0', 'valid': True}, 'e83d0d46-6ea6-4aa3-= 80bf-6e95c66b0454': {'code': 0, 'actual': True, 'version': 0, 'acquired': T= rue, 'delay': '0.00011454', 'lastCheck': '6.0', 'valid': True}, '65ca2e2d-b= 472-4bee-85b4-09a161464b20': {'code': 0, 'actual': True, 'version': 4, 'acq= uired': True, 'delay': '0.000145955', 'lastCheck': '6.0', 'valid': True}, '= 4bf2ba2f-f57a-4d9f-b42a-fb78f440a358': {'code': 0, 'actual': True, 'version= ': 0, 'acquired': True, 'delay': '0.000209871', 'lastCheck': '5.9', 'valid'= : True}, '42e591b7-f86c-4b67-a3d2-40cc007f7662': {'code': 0, 'actual': True= , 'version': 4, 'acquired': True, 'delay': '9.5744e-05', 'lastCheck': '6.0'= , 'valid': True}} from=3Dinternal, task_id=3D20e0b29b-d3cd-4e44-b92c-213f9c= 984ab2 (api:52)<br id=3D"yui_3_16_0_ym19_1_1519085807369_70697">2018-02-20 = 14:41:49,083+0800 INFO&nbsp; (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call H= ost.getAllVmStats succeeded in 0.00 seconds (__init__:539)<br id=3D"yui_3_1= 6_0_ym19_1_1519085807369_70698"></div><div dir=3D"ltr"><br></div><div dir= =3D"ltr" id=3D"yui_3_16_0_ym19_1_1519085807369_70767"><br></div><div dir=3D= "ltr" id=3D"yui_3_16_0_ym19_1_1519085807369_70768">Engine log</div><div dir= =3D"ltr"><br></div><div dir=3D"ltr" id=3D"yui_3_16_0_ym19_1_1519085807369_7= 0769">skId '404ccecc-aa7f-45ea-89e4-726956269bc9' task status 'finished'<br= id=3D"yui_3_16_0_ym19_1_1519085807369_70808">2018-02-20 14:42:21,966+08 IN= FO&nbsp; [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (or= g.ovirt.thread.pool-7-thread-38) [29528f9] spmStart polling ended, spm stat= us: SPM<br id=3D"yui_3_16_0_ym19_1_1519085807369_70809">2018-02-20 14:42:21= ,967+08 INFO&nbsp; [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskV= DSCommand] (org.ovirt.thread.pool-7-thread-38) [29528f9] START, HSMClearTas= kVDSCommand(HostName =3D Node1, HSMTaskGuidBaseVDSCommandParameters:{runAsy= nc=3D'true', hostId=3D'7dee35bb-8c97-4f6a-b6cd-abc4258540e4', taskId=3D'404= ccecc-aa7f-45ea-89e4-726956269bc9'}), log id: 71688f70<br id=3D"yui_3_16_0_= ym19_1_1519085807369_70810">2018-02-20 14:42:22,922+08 INFO&nbsp; [org.ovir= t.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread= .pool-7-thread-38) [29528f9] FINISH, HSMClearTaskVDSCommand, log id: 71688f= 70<br id=3D"yui_3_16_0_ym19_1_1519085807369_70811">2018-02-20 14:42:22,923+= 08 INFO&nbsp; [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand= ] (org.ovirt.thread.pool-7-thread-38) [29528f9] FINISH, SpmStartVDSCommand,= return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7833= 2453, log id: 3ea35d5<br id=3D"yui_3_16_0_ym19_1_1519085807369_70812">2018-= 02-20 14:42:22,935+08 INFO&nbsp; [org.ovirt.engine.core.vdsbroker.irsbroker= .IrsProxy] (org.ovirt.thread.pool-7-thread-38) [29528f9] Initialize Irs pro= xy from vds: dev2node1.lares.com.ph<br id=3D"yui_3_16_0_ym19_1_151908580736= 9_70813">2018-02-20 14:42:22,951+08 INFO&nbsp; [org.ovirt.engine.core.dal.d= bbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-= 38) [29528f9] EVENT_ID: IRS_HOSTED_ON_VDS(204), Correlation ID: null, Call = Stack: null, Custom ID: null, Custom Event ID: -1, Message: Storage Pool Ma= nager runs on Host Node1 (Address: dev2node1.lares.com.ph).<br id=3D"yui_3_= 16_0_ym19_1_1519085807369_70814">2018-02-20 14:42:22,952+08 INFO&nbsp; [org= .ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.o= virt.thread.pool-7-thread-38) [29528f9] -- executeIrsBrokerCommand: Attempt= ing on storage pool '5a865884-0366-0330-02b8-0000000002d4'<br id=3D"yui_3_1= 6_0_ym19_1_1519085807369_70815">2018-02-20 14:42:22,952+08 INFO&nbsp; [org.= ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ov= irt.thread.pool-7-thread-38) [29528f9] START, HSMGetAllTasksInfoVDSCommand(= HostName =3D Node1, VdsIdVDSCommandParametersBase:{runAsync=3D'true', hostI= d=3D'7dee35bb-8c97-4f6a-b6cd-abc4258540e4'}), log id: 1bdbea9d<br id=3D"yui= _3_16_0_ym19_1_1519085807369_70816">2018-02-20 14:42:22,955+08 INFO&nbsp; [= org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (or= g.ovirt.thread.pool-7-thread-7) [29528f9] START, SPMGetAllTasksInfoVDSComma= nd( IrsBaseVDSCommandParameters:{runAsync=3D'true', storagePoolId=3D'5a8658= 84-0366-0330-02b8-0000000002d4', ignoreFailoverLimit=3D'false'}), log id: 5= c2422d6<br id=3D"yui_3_16_0_ym19_1_1519085807369_70817">2018-02-20 14:42:23= ,956+08 INFO&nbsp; [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTask= sInfoVDSCommand] (org.ovirt.thread.pool-7-thread-38) [29528f9] FINISH, HSMG= etAllTasksInfoVDSCommand, return: [], log id: 1bdbea9d<br id=3D"yui_3_16_0_= ym19_1_1519085807369_70818">2018-02-20 14:42:23,956+08 INFO&nbsp; [org.ovir= t.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.= thread.pool-7-thread-38) [29528f9] FINISH, SPMGetAllTasksInfoVDSCommand, re= turn: [], log id: 58cbe7b7<br id=3D"yui_3_16_0_ym19_1_1519085807369_70819">= 2018-02-20 14:42:23,956+08 INFO&nbsp; [org.ovirt.engine.core.bll.tasks.Asyn= cTaskManager] (org.ovirt.thread.pool-7-thread-38) [29528f9] Discovered no t= asks on Storage Pool 'UnsecuredEnv'<br id=3D"yui_3_16_0_ym19_1_151908580736= 9_70820">2018-02-20 14:42:24,936+08 INFO&nbsp; [org.ovirt.vdsm.jsonrpc.clie= nt.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to dev2node1.l= ares.com.ph/10.10.43.2<br id=3D"yui_3_16_0_ym19_1_1519085807369_70821">2018= -02-20 14:42:27,012+08 WARN&nbsp; [org.ovirt.engine.core.vdsbroker.irsbroke= r.IrsProxy] (org.ovirt.thread.pool-7-thread-43) [] Master domain is not in = sync between DB and VDSM. Domain Node1Container marked as master in DB and = not in the storage<br id=3D"yui_3_16_0_ym19_1_1519085807369_70822">2018-02-= 20 14:42:27,026+08 WARN&nbsp; [org.ovirt.engine.core.dal.dbbroker.auditlogh= andling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-43) [] EVENT_ID: = SYSTEM_MASTER_DOMAIN_NOT_IN_SYNC(990), Correlation ID: null, Call Stack: nu= ll, Custom ID: null, Custom Event ID: -1, Message: Sync Error on Master Dom= ain between Host Node1 and oVirt Engine. Domain: Node1Container is marked a= s Master in oVirt Engine database but not on the Storage side. Please consu= lt with Support on how to fix this issue.<br id=3D"yui_3_16_0_ym19_1_151908= 5807369_70823">2018-02-20 14:42:27,103+08 INFO&nbsp; [org.ovirt.engine.core= .bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-= thread-43) [3e5965ca] Running command: ReconstructMasterDomainCommand inter= nal: true. Entities affected :&nbsp; ID: f3e372e3-1251-4195-a4b9-1027e40059= df Type: Storage<br id=3D"yui_3_16_0_ym19_1_1519085807369_70824">2018-02-20= 14:42:27,137+08 INFO&nbsp; [org.ovirt.engine.core.vdsbroker.irsbroker.Rese= tIrsVDSCommand] (org.ovirt.thread.pool-7-thread-43) [3e5965ca] START, Reset= IrsVDSCommand( ResetIrsVDSCommandParameters:{runAsync=3D'true', storagePool= Id=3D'5a865884-0366-0330-02b8-0000000002d4', ignoreFailoverLimit=3D'false',= vdsId=3D'7dee35bb-8c97-4f6a-b6cd-abc4258540e4', ignoreStopFailed=3D'true'}= ), log id: 3e0a239d<br id=3D"yui_3_16_0_ym19_1_1519085807369_70825">2018-02= -20 14:42:27,140+08 INFO&nbsp; [org.ovirt.engine.core.vdsbroker.vdsbroker.S= pmStopVDSCommand] (org.ovirt.thread.pool-7-thread-43) [3e5965ca] START, Spm= StopVDSCommand(HostName =3D Node1, SpmStopVDSCommandParameters:{runAsync=3D= 'true', hostId=3D'7dee35bb-8c97-4f6a-b6cd-abc4258540e4', storagePoolId=3D'5= a865884-0366-0330-02b8-0000000002d4'}), log id: 7c67bf06<br id=3D"yui_3_16_= 0_ym19_1_1519085807369_70826">2018-02-20 14:42:28,144+08 INFO&nbsp; [org.ov= irt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.po= ol-7-thread-43) [3e5965ca] SpmStopVDSCommand::Stopping SPM on vds 'Node1', = pool id '5a865884-0366-0330-02b8-0000000002d4'<br></div><div id=3D"yui_3_16= _0_ym19_1_1519085807369_70529"><span></span></div> <div class=3D"qtdSeparat= eBR"><br><br></div><div class=3D"yahoo_quoted" style=3D"display: block;"> <= div style=3D"font-family: Helvetica Neue, Helvetica, Arial, Lucida Grande, = sans-serif; font-size: 10px;"> <div style=3D"font-family: HelveticaNeue, He= lvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 16px;= "> <div dir=3D"ltr"><font size=3D"2" face=3D"Arial"> On Tuesday, February 2= 0, 2018 2:33 PM, michael pagdanganan &amp;lt;mhke_aj5566(a)yahoo.com&amp;gt; wrote:<b= r></font></div> <br><br> <div class=3D"y_msg_container"><div id=3D"yiv0115= 319588"><div><div style=3D"color:#000;background-color:#fff;font-family:Hel= vetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:10px;"><= div id=3D"yiv0115319588yui_3_16_0_ym19_1_1519085807369_60672"><font size=3D= "3">Thanks for quick response,</font></div><div id=3D"yiv0115319588yui_3_16= _0_ym19_1_1519085807369_62803"><font size=3D"3"><br clear=3D"none"></font><= /div><div id=3D"yiv0115319588yui_3_16_0_ym19_1_1519085807369_64423"><font s= ize=3D"3">see attachment.<br clear=3D"none"></font></div><div id=3D"yiv0115= 319588yui_3_16_0_ym19_1_1519085807369_60552"><span></span></div> <div class= =3D"yiv0115319588qtdSeparateBR"><br clear=3D"none"><br clear=3D"none"></div= yle=3D"font-family:Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-se= rif;font-size:10px;"> <div style=3D"font-family:HelveticaNeue, Helvetica Ne= ue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px;"> <div clas= s=3D"yiv0115319588yqt2980525560" id=3D"yiv0115319588yqtfd88886"><div dir=3D= "ltr"><font size=3D"2" face=3D"Arial"> On Tuesday, February 20, 2018 2:10 P= M, Eyal Shenitzky &amp;lt;eshenitz(a)redhat.com&amp;gt; wrote:<br clear=3D"none"></fo= nt></div> <br clear=3D"none"><br clear=3D"none"> <div class=3D"yiv01153195= 88y_msg_container"><div id=3D"yiv0115319588"><div><div dir=3D"ltr">Hi,&nbsp= ;<div><br clear=3D"none"></div><div>Can you please attach full Engine and V= DSM logs?</div><div><br clear=3D"none"></div><div>Thanks,</div></div><div c= lass=3D"yiv0115319588gmail_extra"><br clear=3D"none"><div class=3D"yiv01153= 19588gmail_quote">On Tue, Feb 20, 2018 at 2:59 AM, michael pagdanganan <spa= n dir=3D"ltr">&lt;<a rel=3D"nofollow" shape=3D"rect" ymailto=3D"mailto:mhke= _aj5566(a)yahoo.com&quot; target=3D"_blank" href=3D"mailto:mhke_aj5566@yahoo.com">= mhke_aj5566(a)yahoo.com&lt;/a&gt;&amp;gt;&lt;/span&gt; wrote:<br clear=3D"none"><blockquote c= lass=3D"yiv0115319588gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex;"><div class=3D"yiv0115319588yqt7135196070" i= d=3D"yiv0115319588yqt29383"><div><div style=3D"color:#000;background-color:= #fff;font-family:Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-seri= f;font-size:10px;"><div dir=3D"ltr" id=3D"yiv0115319588m_-46053014922747698= 28yui_3_16_0_ym19_1_1519085807369_10542"><font id=3D"yiv0115319588m_-460530= 1492274769828yui_3_16_0_ym19_1_1519085807369_15967" size=3D"3">My storage p= ool has 2 Master Domain(</font><font id=3D"yiv0115319588m_-4605301492274769= 828yui_3_16_0_ym19_1_1519085807369_16063" size=3D"3"><font id=3D"yiv0115319= 588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_15968" size=3D"3">= Stored2 on Node2,Node1Container on Node1)</font>. and my old master domain = </font><font id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_151= 9085807369_15934" size=3D"3"><font id=3D"yiv0115319588m_-460530149227476982= 8yui_3_16_0_ym19_1_1519085807369_16064" size=3D"3">(</font><font id=3D"yiv0= 115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_16065" size= =3D"3"><font id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_151= 9085807369_16066" size=3D"3">DATANd01 on Node1 )</font></font> hung on prep= aring for maintenance. When I tried to activate my old master domain (</fon= t><font id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_15190858= 07369_10517" size=3D"3"><font id=3D"yiv0115319588m_-4605301492274769828yui_= 3_16_0_ym19_1_1519085807369_15935" size=3D"3">DATANd01 on Node1 )</font> al= l&nbsp; storage domain goes down and up master keep on rotating.</font></di= v><div id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_151908580= 7369_10905"><font id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_= 1_1519085807369_10517" size=3D"3"><br clear=3D"none"></font></div><div id= =3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_1069= 2"><font id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085= 807369_10517" size=3D"3"><br clear=3D"none"></font></div><div dir=3D"ltr" i= d=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_107= 10"><font id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_151908= 5807369_10517" size=3D"3">Ovirt Version (oVirt Engine Version: 4.1.9.1.el7.= centos)<br clear=3D"none"></font></div><div dir=3D"ltr" id=3D"yiv0115319588= m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_14202"><font id=3D"yi= v0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" siz= e=3D"3"><br clear=3D"none"></font></div><div dir=3D"ltr" id=3D"yiv011531958= 8m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_12604"><font id=3D"y= iv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" si= ze=3D"3"><br clear=3D"none"></font></div><div dir=3D"ltr" id=3D"yiv01153195= 88m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_14203"><font id=3D"= yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" s= ize=3D"3">Event Error:</font></div><div dir=3D"ltr" id=3D"yiv0115319588m_-4= 605301492274769828yui_3_16_0_ym19_1_1519085807369_14215"><font id=3D"yiv011= 5319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size=3D= "3">Sync Error on Master Domain between Host Node2 and oVirt Engine. Domain= Stored2 is marked as master in ovirt engine Database but not on storage si= de Please consult with support<br id=3D"yiv0115319588m_-4605301492274769828= yui_3_16_0_ym19_1_1519085807369_12607" clear=3D"none"></font></div><div dir= =3D"ltr" id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085= 807369_14204"><font id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym1= 9_1_1519085807369_10517" size=3D"3"><br clear=3D"none"></font></div><div di= r=3D"ltr" id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_151908= 5807369_15891"><font id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym= 19_1_1519085807369_10517" size=3D"3">VDSM Node2 command ConnectStoragePoolV= DS failed: Wrong Master domain or it's version: u'SD=3Df3e372e3-1251-4195-a= 4b9- 1027e40059df, pool=3D5a865884-0366-0330-02b8- 0000</font></div><div di= r=3D"ltr" id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_151908= 5807369_15895"><font id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym= 19_1_1519085807369_10517" size=3D"3"><br clear=3D"none"></font></div><div d= ir=3D"ltr" id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_15190= 85807369_15896"><font id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_y= m19_1_1519085807369_10517" size=3D"3">VDSM Node2 command HSMGetAllTastsStat= usesVDS failed: Not SPM: ()<br id=3D"yiv0115319588m_-4605301492274769828yui= _3_16_0_ym19_1_1519085807369_15901" clear=3D"none"><br id=3D"yiv0115319588m= _-4605301492274769828yui_3_16_0_ym19_1_1519085807369_15902" clear=3D"none">= Failed to deactivate Storage Domain DATANd01 (Data Center UnsecuredEnv)<br = clear=3D"none"></font></div><div id=3D"yiv0115319588m_-4605301492274769828y= ui_3_16_0_ym19_1_1519085807369_10545"><font id=3D"yiv0115319588m_-460530149= 2274769828yui_3_16_0_ym19_1_1519085807369_10517" size=3D"3"><br clear=3D"no= ne"></font></div><div id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_y= m19_1_1519085807369_10551"><font id=3D"yiv0115319588m_-4605301492274769828y= ui_3_16_0_ym19_1_1519085807369_10517" size=3D"3">Here's logs from engine:</= font></div><div id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_= 1519085807369_10620"><font id=3D"yiv0115319588m_-4605301492274769828yui_3_1= 6_0_ym19_1_1519085807369_10517" size=3D"3"><br clear=3D"none"></font></div>= <div id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_15190858073= 69_10573"><font id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_= 1519085807369_10517" size=3D"3">------------------------------ ------------= ------------------ ------------------------------ -------------</font></div= 1_1519085807369_10588"><font id=3D"yiv0115319588m_-4605301492274769828yui_3= _16_0_ym19_1_1519085807369_10517" size=3D"3">[root@dev2engine ~]# tail /var= /log/messages<br id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1= _1519085807369_10609" clear=3D"none">Feb 20 07:01:01 dev2engine systemd: St= arting Session 20 of user root.<br id=3D"yiv0115319588m_-460530149227476982= 8yui_3_16_0_ym19_1_1519085807369_10610" clear=3D"none">Feb 20 07:01:01 dev2= engine systemd: Removed slice User Slice of root.<br id=3D"yiv0115319588m_-= 4605301492274769828yui_3_16_0_ym19_1_1519085807369_10611" clear=3D"none">Fe= b 20 07:01:01 dev2engine systemd: Stopping User Slice of root.<br id=3D"yiv= 0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10612" clea= r=3D"none">Feb 20 07:58:52 dev2engine systemd: Created slice User Slice of = root.<br id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085= 807369_10613" clear=3D"none">Feb 20 07:58:52 dev2engine systemd: Starting U= ser Slice of root.<br id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_y= m19_1_1519085807369_10614" clear=3D"none">Feb 20 07:58:52 dev2engine system= d-logind: New session 21 of user root.<br id=3D"yiv0115319588m_-46053014922= 74769828yui_3_16_0_ym19_1_1519085807369_10615" clear=3D"none">Feb 20 07:58:= 52 dev2engine systemd: Started Session 21 of user root.<br id=3D"yiv0115319= 588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10616" clear=3D"no= ne">Feb 20 07:58:52 dev2engine systemd: Starting Session 21 of user root.<b= r id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_= 10617" clear=3D"none">Feb 20 08:01:01 dev2engine systemd: Started Session 2= 2 of user root.<br id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19= _1_1519085807369_10618" clear=3D"none">Feb 20 08:01:01 dev2engine systemd: = Starting Session 22 of user root.</font></div><div dir=3D"ltr" id=3D"yiv011= 5319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10619"><font i= d=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_105= 17" size=3D"3"><br clear=3D"none"></font></div><div dir=3D"ltr" id=3D"yiv01= 15319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10623"><br cl= ear=3D"none"><font id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19= _1_1519085807369_10517" size=3D"3"><font id=3D"yiv0115319588m_-460530149227= 4769828yui_3_16_0_ym19_1_1519085807369_10626" size=3D"3">------------------= ------------ ------------------------------ ------------------------------ = -------------</font></font></div><div dir=3D"ltr" id=3D"yiv0115319588m_-460= 5301492274769828yui_3_16_0_ym19_1_1519085807369_10654"><font id=3D"yiv01153= 19588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size=3D"3= "><font id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_15190858= 07369_10626" size=3D"3">[root@dev2engine ~]# tail /var/log/ovirt-engine/eng= ine. log<br id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519= 085807369_10677" clear=3D"none">2018-02-20 08:01:16,062+08 INFO&nbsp; [org.= ovirt.engine.core.bll. eventqueue.EventQueueMonitor] (org.ovirt.thread.pool= -7- thread-32) [102e9d3c] Finished reconstruct for pool '5a865884-0366-0330= -02b8- 0000000002d4'. Clearing event queue<br id=3D"yiv0115319588m_-4605301= 492274769828yui_3_16_0_ym19_1_1519085807369_10678" clear=3D"none">2018-02-2= 0 08:01:27,825+08 WARN&nbsp; [org.ovirt.engine.core. vdsbroker.irsbroker.Ir= sProxy] (org.ovirt.thread.pool-7- thread-23) [] Master domain is not in syn= c between DB and VDSM. Domain Stored2 marked as master in DB and not in the= storage<br id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519= 085807369_10679" clear=3D"none">2018-02-20 08:01:27,862+08 WARN&nbsp; [org.= ovirt.engine.core.bll. storage.pool. ReconstructMasterDomainCommand ] (org.= ovirt.thread.pool-7- thread-23) [213f42b9] Validation of action 'Reconstruc= tMasterDomain' failed for user SYSTEM. Reasons: VAR__ACTION__RECONSTRUCT_ M= ASTER,VAR__TYPE__STORAGE__ DOMAIN,ACTION_TYPE_FAILED_ STORAGE_DOMAIN_STATUS= _ ILLEGAL2,$status PreparingForMaintenance<br id=3D"yiv0115319588m_-4605301= 492274769828yui_3_16_0_ym19_1_1519085807369_10680" clear=3D"none">2018-02-2= 0 08:01:27,882+08 INFO&nbsp; [org.ovirt.engine.core.bll. eventqueue.EventQu= eueMonitor] (org.ovirt.thread.pool-7- thread-20) [929330e] Finished reconst= ruct for pool '5a865884-0366-0330-02b8- 0000000002d4'. Clearing event queue= <br id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_151908580736= 9_10681" clear=3D"none">2018-02-20 08:01:40,106+08 WARN&nbsp; [org.ovirt.en= gine.core. vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-7- thread-1= 7) [] Master domain is not in sync between DB and VDSM. Domain Stored2 mark= ed as master in DB and not in the storage<br id=3D"yiv0115319588m_-46053014= 92274769828yui_3_16_0_ym19_1_1519085807369_10682" clear=3D"none">2018-02-20= 08:01:40,197+08 WARN&nbsp; [org.ovirt.engine.core.bll. storage.pool. Recon= structMasterDomainCommand ] (org.ovirt.thread.pool-7- thread-17) [7af552c1]= Validation of action 'ReconstructMasterDomain' failed for user SYSTEM. Rea= sons: VAR__ACTION__RECONSTRUCT_ MASTER,VAR__TYPE__STORAGE__ DOMAIN,ACTION_T= YPE_FAILED_ STORAGE_DOMAIN_STATUS_ ILLEGAL2,$status PreparingForMaintenance= <br id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_151908580736= 9_10683" clear=3D"none">2018-02-20 08:01:40,246+08 INFO&nbsp; [org.ovirt.en= gine.core.bll. eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7- thre= ad-22) [73673040] Finished reconstruct for pool '5a865884-0366-0330-02b8- 0= 000000002d4'. Clearing event queue<br id=3D"yiv0115319588m_-460530149227476= 9828yui_3_16_0_ym19_1_1519085807369_10684" clear=3D"none">2018-02-20 08:01:= 51,809+08 WARN&nbsp; [org.ovirt.engine.core. vdsbroker.irsbroker.IrsProxy] = (org.ovirt.thread.pool-7- thread-26) [] Master domain is not in sync betwee= n DB and VDSM. Domain Stored2 marked as master in DB and not in the storage= <br id=3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_151908580736= 9_10685" clear=3D"none">2018-02-20 08:01:51,846+08 WARN&nbsp; [org.ovirt.en= gine.core.bll. storage.pool. ReconstructMasterDomainCommand ] (org.ovirt.th= read.pool-7- thread-26) [20307cbe] Validation of action 'ReconstructMasterD= omain' failed for user SYSTEM. Reasons: VAR__ACTION__RECONSTRUCT_ MASTER,VA= R__TYPE__STORAGE__ DOMAIN,ACTION_TYPE_FAILED_ STORAGE_DOMAIN_STATUS_ ILLEGA= L2,$status PreparingForMaintenance<br id=3D"yiv0115319588m_-460530149227476= 9828yui_3_16_0_ym19_1_1519085807369_10686" clear=3D"none">2018-02-20 08:01:= 51,866+08 INFO&nbsp; [org.ovirt.engine.core.bll. eventqueue.EventQueueMonit= or] (org.ovirt.thread.pool-7- thread-49) [2c11a866] Finished reconstruct fo= r pool '5a865884-0366-0330-02b8- 0000000002d4'. Clearing event queue<br id= =3D"yiv0115319588m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_1068= 7" clear=3D"none"><br clear=3D"none"></font></font></div></div></div></div>= <br clear=3D"none">______________________________ _________________<br clea= r=3D"none"> Users mailing list<br clear=3D"none"> <a rel=3D"nofollow" shape=3D"rect" ymailto=3D"mailto:Users@ovirt.org" targe= t=3D"_blank" href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a><br clear= =3D"none"> <a rel=3D"nofollow" shape=3D"rect" target=3D"_blank" href=3D"http://lists.o= virt.org/mailman/listinfo/users">http://lists.ovirt.org/ mailman/listinfo/u= sers</a><br clear=3D"none"> <br clear=3D"none"></blockquote></div><br clear=3D"none"><br clear=3D"all">= <div><br clear=3D"none"></div>-- <br clear=3D"none"><div class=3D"yiv011531= 9588gmail_signature"><div dir=3D"ltr">Regards,<div>Eyal Shenitzky</div></di= v></div> </div></div></div><br clear=3D"none"><br clear=3D"none"></div> </div></div= </div></div><div class=3D"yiv0115319588yqt2980525560" id=3D"yiv0115319588yq= tfd67072"> </div></div></div></div></div><br><br></div> </div> </div> </= div></div></body></html> ------=_Part_2010355_281080504.1519108987392--