------=_Part_1848497_1716892283.1519088345205 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable My storage pool has 2 Master Domain(Stored2 on Node2,Node1Container on Node= 1). and my old master domain (DATANd01 on Node1 ) hung on preparing for mai= ntenance. When I tried to activate my old master domain (DATANd01 on Node1 = ) all=C2=A0 storage domain goes down and up master keep on rotating. Ovirt Version (oVirt Engine Version: 4.1.9.1.el7.centos) Event Error:Sync Error on Master Domain between Host Node2 and oVirt Engine= . Domain Stored2 is marked as master in ovirt engine Database but not on st= orage side Please consult with support VDSM Node2 command ConnectStoragePoolVDS failed: Wrong Master domain or it'= s version: u'SD=3Df3e372e3-1251-4195-a4b9-1027e40059df, pool=3D5a865884-036= 6-0330-02b8-0000 VDSM Node2 command HSMGetAllTastsStatusesVDS failed: Not SPM: () Failed to deactivate Storage Domain DATANd01 (Data Center UnsecuredEnv) Here's logs from engine: ---------------------------------------------------------------------------= ----------------------------[root@dev2engine ~]# tail /var/log/messages Feb 20 07:01:01 dev2engine systemd: Starting Session 20 of user root. Feb 20 07:01:01 dev2engine systemd: Removed slice User Slice of root. Feb 20 07:01:01 dev2engine systemd: Stopping User Slice of root. Feb 20 07:58:52 dev2engine systemd: Created slice User Slice of root. Feb 20 07:58:52 dev2engine systemd: Starting User Slice of root. Feb 20 07:58:52 dev2engine systemd-logind: New session 21 of user root. Feb 20 07:58:52 dev2engine systemd: Started Session 21 of user root. Feb 20 07:58:52 dev2engine systemd: Starting Session 21 of user root. Feb 20 08:01:01 dev2engine systemd: Started Session 22 of user root. Feb 20 08:01:01 dev2engine systemd: Starting Session 22 of user root. ---------------------------------------------------------------------------= ----------------------------[root@dev2engine ~]# tail /var/log/ovirt-engine= /engine.log 2018-02-20 08:01:16,062+08 INFO=C2=A0 [org.ovirt.engine.core.bll.eventqueue= .EventQueueMonitor] (org.ovirt.thread.pool-7-thread-32) [102e9d3c] Finished= reconstruct for pool '5a865884-0366-0330-02b8-0000000002d4'. Clearing even= t queue 2018-02-20 08:01:27,825+08 WARN=C2=A0 [org.ovirt.engine.core.vdsbroker.irsb= roker.IrsProxy] (org.ovirt.thread.pool-7-thread-23) [] Master domain is not= in sync between DB and VDSM. Domain Stored2 marked as master in DB and not= in the storage 2018-02-20 08:01:27,862+08 WARN=C2=A0 [org.ovirt.engine.core.bll.storage.po= ol.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-23) [213= f42b9] Validation of action 'ReconstructMasterDomain' failed for user SYSTE= M. Reasons: VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTI= ON_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL2,$status PreparingForMaintenan= ce 2018-02-20 08:01:27,882+08 INFO=C2=A0 [org.ovirt.engine.core.bll.eventqueue= .EventQueueMonitor] (org.ovirt.thread.pool-7-thread-20) [929330e] Finished = reconstruct for pool '5a865884-0366-0330-02b8-0000000002d4'. Clearing event= queue 2018-02-20 08:01:40,106+08 WARN=C2=A0 [org.ovirt.engine.core.vdsbroker.irsb= roker.IrsProxy] (org.ovirt.thread.pool-7-thread-17) [] Master domain is not= in sync between DB and VDSM. Domain Stored2 marked as master in DB and not= in the storage 2018-02-20 08:01:40,197+08 WARN=C2=A0 [org.ovirt.engine.core.bll.storage.po= ol.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-17) [7af= 552c1] Validation of action 'ReconstructMasterDomain' failed for user SYSTE= M. Reasons: VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTI= ON_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL2,$status PreparingForMaintenan= ce 2018-02-20 08:01:40,246+08 INFO=C2=A0 [org.ovirt.engine.core.bll.eventqueue= .EventQueueMonitor] (org.ovirt.thread.pool-7-thread-22) [73673040] Finished= reconstruct for pool '5a865884-0366-0330-02b8-0000000002d4'. Clearing even= t queue 2018-02-20 08:01:51,809+08 WARN=C2=A0 [org.ovirt.engine.core.vdsbroker.irsb= roker.IrsProxy] (org.ovirt.thread.pool-7-thread-26) [] Master domain is not= in sync between DB and VDSM. Domain Stored2 marked as master in DB and not= in the storage 2018-02-20 08:01:51,846+08 WARN=C2=A0 [org.ovirt.engine.core.bll.storage.po= ol.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-26) [203= 07cbe] Validation of action 'ReconstructMasterDomain' failed for user SYSTE= M. Reasons: VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTI= ON_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL2,$status PreparingForMaintenan= ce 2018-02-20 08:01:51,866+08 INFO=C2=A0 [org.ovirt.engine.core.bll.eventqueue= .EventQueueMonitor] (org.ovirt.thread.pool-7-thread-49) [2c11a866] Finished= reconstruct for pool '5a865884-0366-0330-02b8-0000000002d4'. Clearing even= t queue ------=_Part_1848497_1716892283.1519088345205 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <html><head></head><body><div style=3D"color:#000; background-color:#fff; f= ont-family:Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font= -size:10px"><div id=3D"yui_3_16_0_ym19_1_1519085807369_10542" dir=3D"ltr"><= font id=3D"yui_3_16_0_ym19_1_1519085807369_15967" size=3D"3">My storage poo= l has 2 Master Domain(</font><font id=3D"yui_3_16_0_ym19_1_1519085807369_16= 063" size=3D"3"><font id=3D"yui_3_16_0_ym19_1_1519085807369_15968" size=3D"= 3">Stored2 on Node2,Node1Container on Node1)</font>. and my old master doma= in </font><font id=3D"yui_3_16_0_ym19_1_1519085807369_15934" size=3D"3"><fo= nt id=3D"yui_3_16_0_ym19_1_1519085807369_16064" size=3D"3">(</font><font id= =3D"yui_3_16_0_ym19_1_1519085807369_16065" size=3D"3"><font id=3D"yui_3_16_= 0_ym19_1_1519085807369_16066" size=3D"3">DATANd01 on Node1 )</font></font> = hung on preparing for maintenance. When I tried to activate my old master d= omain (</font><font id=3D"yui_3_16_0_ym19_1_1519085807369_10517" size=3D"3"= Node1 )</font> all&nbsp; storage domain goes down and up master keep on rot= ating.</font></div><div id=3D"yui_3_16_0_ym19_1_1519085807369_10905"><font = id=3D"yui_3_16_0_ym19_1_1519085807369_10517" size=3D"3"><br></font></div><d= iv id=3D"yui_3_16_0_ym19_1_1519085807369_10692"><font id=3D"yui_3_16_0_ym19= _1_1519085807369_10517" size=3D"3"><br></font></div><div id=3D"yui_3_16_0_y= m19_1_1519085807369_10710" dir=3D"ltr"><font id=3D"yui_3_16_0_ym19_1_151908= 5807369_10517" size=3D"3">Ovirt Version (oVirt Engine Version: 4.1.9.1.el7.= centos)<br></font></div><div dir=3D"ltr" id=3D"yui_3_16_0_ym19_1_1519085807= 369_14202"><font id=3D"yui_3_16_0_ym19_1_1519085807369_10517" size=3D"3"><b= r></font></div><div dir=3D"ltr" id=3D"yui_3_16_0_ym19_1_1519085807369_12604= "><font id=3D"yui_3_16_0_ym19_1_1519085807369_10517" size=3D"3"><br></font>= </div><div dir=3D"ltr" id=3D"yui_3_16_0_ym19_1_1519085807369_14203"><font i= d=3D"yui_3_16_0_ym19_1_1519085807369_10517" size=3D"3">Event Error:</font><= /div><div dir=3D"ltr" id=3D"yui_3_16_0_ym19_1_1519085807369_14215"><font id= =3D"yui_3_16_0_ym19_1_1519085807369_10517" size=3D"3">Sync Error on Master = Domain between Host Node2 and oVirt Engine. Domain Stored2 is marked as mas= ter in ovirt engine Database but not on storage side Please consult with su= pport<br id=3D"yui_3_16_0_ym19_1_1519085807369_12607"></font></div><div dir= =3D"ltr" id=3D"yui_3_16_0_ym19_1_1519085807369_14204"><font id=3D"yui_3_16_= 0_ym19_1_1519085807369_10517" size=3D"3"><br></font></div><div dir=3D"ltr" = id=3D"yui_3_16_0_ym19_1_1519085807369_15891"><font id=3D"yui_3_16_0_ym19_1_= 1519085807369_10517" size=3D"3">VDSM Node2 command ConnectStoragePoolVDS fa= iled: Wrong Master domain or it's version: u'SD=3Df3e372e3-1251-4195-a4b9-1= 027e40059df, pool=3D5a865884-0366-0330-02b8-0000</font></div><div dir=3D"lt= r" id=3D"yui_3_16_0_ym19_1_1519085807369_15895"><font id=3D"yui_3_16_0_ym19= _1_1519085807369_10517" size=3D"3"><br></font></div><div dir=3D"ltr" id=3D"= yui_3_16_0_ym19_1_1519085807369_15896"><font id=3D"yui_3_16_0_ym19_1_151908= 5807369_10517" size=3D"3">VDSM Node2 command HSMGetAllTastsStatusesVDS fail= ed: Not SPM: ()<br id=3D"yui_3_16_0_ym19_1_1519085807369_15901"><br id=3D"y= ui_3_16_0_ym19_1_1519085807369_15902">Failed to deactivate Storage Domain D= ATANd01 (Data Center UnsecuredEnv)<br></font></div><div id=3D"yui_3_16_0_ym= 19_1_1519085807369_10545"><font id=3D"yui_3_16_0_ym19_1_1519085807369_10517= " size=3D"3"><br></font></div><div id=3D"yui_3_16_0_ym19_1_1519085807369_10= 551"><font id=3D"yui_3_16_0_ym19_1_1519085807369_10517" size=3D"3">Here's l= ogs from engine:</font></div><div id=3D"yui_3_16_0_ym19_1_1519085807369_106= 20"><font id=3D"yui_3_16_0_ym19_1_1519085807369_10517" size=3D"3"><br></fon= t></div><div id=3D"yui_3_16_0_ym19_1_1519085807369_10573"><font id=3D"yui_3= _16_0_ym19_1_1519085807369_10517" size=3D"3">------------------------------= -------------------------------------------------------------------------</= font></div><div id=3D"yui_3_16_0_ym19_1_1519085807369_10588" dir=3D"ltr"><f= ont id=3D"yui_3_16_0_ym19_1_1519085807369_10517" size=3D"3">[root@dev2engin= e ~]# tail /var/log/messages<br id=3D"yui_3_16_0_ym19_1_1519085807369_10609= ">Feb 20 07:01:01 dev2engine systemd: Starting Session 20 of user root.<br = id=3D"yui_3_16_0_ym19_1_1519085807369_10610">Feb 20 07:01:01 dev2engine sys= temd: Removed slice User Slice of root.<br id=3D"yui_3_16_0_ym19_1_15190858= 07369_10611">Feb 20 07:01:01 dev2engine systemd: Stopping User Slice of roo= t.<br id=3D"yui_3_16_0_ym19_1_1519085807369_10612">Feb 20 07:58:52 dev2engi= ne systemd: Created slice User Slice of root.<br id=3D"yui_3_16_0_ym19_1_15= 19085807369_10613">Feb 20 07:58:52 dev2engine systemd: Starting User Slice = of root.<br id=3D"yui_3_16_0_ym19_1_1519085807369_10614">Feb 20 07:58:52 de= v2engine systemd-logind: New session 21 of user root.<br id=3D"yui_3_16_0_y= m19_1_1519085807369_10615">Feb 20 07:58:52 dev2engine systemd: Started Sess= ion 21 of user root.<br id=3D"yui_3_16_0_ym19_1_1519085807369_10616">Feb 20= 07:58:52 dev2engine systemd: Starting Session 21 of user root.<br id=3D"yu= i_3_16_0_ym19_1_1519085807369_10617">Feb 20 08:01:01 dev2engine systemd: St= arted Session 22 of user root.<br id=3D"yui_3_16_0_ym19_1_1519085807369_106= 18">Feb 20 08:01:01 dev2engine systemd: Starting Session 22 of user root.</= font></div><div dir=3D"ltr" id=3D"yui_3_16_0_ym19_1_1519085807369_10619"><f= ont id=3D"yui_3_16_0_ym19_1_1519085807369_10517" size=3D"3"><br></font></di= v><div dir=3D"ltr" id=3D"yui_3_16_0_ym19_1_1519085807369_10623"><br><font i= d=3D"yui_3_16_0_ym19_1_1519085807369_10517" size=3D"3"><font id=3D"yui_3_16= _0_ym19_1_1519085807369_10626" size=3D"3">---------------------------------= ----------------------------------------------------------------------</fon= t></font></div><div dir=3D"ltr" id=3D"yui_3_16_0_ym19_1_1519085807369_10654= "><font id=3D"yui_3_16_0_ym19_1_1519085807369_10517" size=3D"3"><font id=3D= "yui_3_16_0_ym19_1_1519085807369_10626" size=3D"3">[root@dev2engine ~]# tai= l /var/log/ovirt-engine/engine.log<br id=3D"yui_3_16_0_ym19_1_1519085807369= _10677">2018-02-20 08:01:16,062+08 INFO&nbsp; [org.ovirt.engine.core.bll.ev= entqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-32) [102e9d3c] = Finished reconstruct for pool '5a865884-0366-0330-02b8-0000000002d4'. Clear= ing event queue<br id=3D"yui_3_16_0_ym19_1_1519085807369_10678">2018-02-20 = 08:01:27,825+08 WARN&nbsp; [org.ovirt.engine.core.vdsbroker.irsbroker.IrsPr= oxy] (org.ovirt.thread.pool-7-thread-23) [] Master domain is not in sync be= tween DB and VDSM. Domain Stored2 marked as master in DB and not in the sto= rage<br id=3D"yui_3_16_0_ym19_1_1519085807369_10679">2018-02-20 08:01:27,86= 2+08 WARN&nbsp; [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDo= mainCommand] (org.ovirt.thread.pool-7-thread-23) [213f42b9] Validation of a= ction 'ReconstructMasterDomain' failed for user SYSTEM. Reasons: VAR__ACTIO= N__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE= _DOMAIN_STATUS_ILLEGAL2,$status PreparingForMaintenance<br id=3D"yui_3_16_0= _ym19_1_1519085807369_10680">2018-02-20 08:01:27,882+08 INFO&nbsp; [org.ovi= rt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-t= hread-20) [929330e] Finished reconstruct for pool '5a865884-0366-0330-02b8-= 0000000002d4'. Clearing event queue<br id=3D"yui_3_16_0_ym19_1_151908580736= 9_10681">2018-02-20 08:01:40,106+08 WARN&nbsp; [org.ovirt.engine.core.vdsbr= oker.irsbroker.IrsProxy] (org.ovirt.thread.pool-7-thread-17) [] Master doma= in is not in sync between DB and VDSM. Domain Stored2 marked as master in D= B and not in the storage<br id=3D"yui_3_16_0_ym19_1_1519085807369_10682">20= 18-02-20 08:01:40,197+08 WARN&nbsp; [org.ovirt.engine.core.bll.storage.pool= .ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-17) [7af55= 2c1] Validation of action 'ReconstructMasterDomain' failed for user SYSTEM.= Reasons: VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION= _TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL2,$status PreparingForMaintenance= <br id=3D"yui_3_16_0_ym19_1_1519085807369_10683">2018-02-20 08:01:40,246+08= INFO&nbsp; [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.o= virt.thread.pool-7-thread-22) [73673040] Finished reconstruct for pool '5a8= 65884-0366-0330-02b8-0000000002d4'. Clearing event queue<br id=3D"yui_3_16_= 0_ym19_1_1519085807369_10684">2018-02-20 08:01:51,809+08 WARN&nbsp; [org.ov= irt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-7-thre= ad-26) [] Master domain is not in sync between DB and VDSM. Domain Stored2 = marked as master in DB and not in the storage<br id=3D"yui_3_16_0_ym19_1_15= 19085807369_10685">2018-02-20 08:01:51,846+08 WARN&nbsp; [org.ovirt.engine.= core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.poo= l-7-thread-26) [20307cbe] Validation of action 'ReconstructMasterDomain' fa= iled for user SYSTEM. Reasons: VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__S= TORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL2,$status Pr= eparingForMaintenance<br id=3D"yui_3_16_0_ym19_1_1519085807369_10686">2018-= 02-20 08:01:51,866+08 INFO&nbsp; [org.ovirt.engine.core.bll.eventqueue.Even= tQueueMonitor] (org.ovirt.thread.pool-7-thread-49) [2c11a866] Finished reco= nstruct for pool '5a865884-0366-0330-02b8-0000000002d4'. Clearing event que= ue<br id=3D"yui_3_16_0_ym19_1_1519085807369_10687"><br></font></font></div>= </div></body></html> ------=_Part_1848497_1716892283.1519088345205--