<html><head></head><body><div style="color:#000; background-color:#fff; font-family:Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:10px"><div id="yui_3_16_0_ym19_1_1519085807369_10542" dir="ltr"><font id="yui_3_16_0_ym19_1_1519085807369_15967" size="3">My storage pool has 2 Master Domain(</font><font id="yui_3_16_0_ym19_1_1519085807369_16063" size="3"><font id="yui_3_16_0_ym19_1_1519085807369_15968" size="3">Stored2 on Node2,Node1Container on Node1)</font>. and my old master domain </font><font id="yui_3_16_0_ym19_1_1519085807369_15934" size="3"><font id="yui_3_16_0_ym19_1_1519085807369_16064" size="3">(</font><font id="yui_3_16_0_ym19_1_1519085807369_16065" size="3"><font id="yui_3_16_0_ym19_1_1519085807369_16066" size="3">DATANd01 on Node1 )</font></font> hung on preparing for maintenance. When I tried to activate my old master domain (</font><font id="yui_3_16_0_ym19_1_1519085807369_10517" size="3"><font id="yui_3_16_0_ym19_1_1519085807369_15935" size="3">DATANd01 on Node1 )</font> all storage domain goes down and up master keep on rotating.</font></div><div id="yui_3_16_0_ym19_1_1519085807369_10905"><font id="yui_3_16_0_ym19_1_1519085807369_10517" size="3"><br></font></div><div id="yui_3_16_0_ym19_1_1519085807369_10692"><font id="yui_3_16_0_ym19_1_1519085807369_10517" size="3"><br></font></div><div id="yui_3_16_0_ym19_1_1519085807369_10710" dir="ltr"><font id="yui_3_16_0_ym19_1_1519085807369_10517" size="3">Ovirt Version (oVirt Engine Version: 4.1.9.1.el7.centos)<br></font></div><div dir="ltr" id="yui_3_16_0_ym19_1_1519085807369_14202"><font id="yui_3_16_0_ym19_1_1519085807369_10517" size="3"><br></font></div><div dir="ltr" id="yui_3_16_0_ym19_1_1519085807369_12604"><font id="yui_3_16_0_ym19_1_1519085807369_10517" size="3"><br></font></div><div dir="ltr" id="yui_3_16_0_ym19_1_1519085807369_14203"><font id="yui_3_16_0_ym19_1_1519085807369_10517" size="3">Event Error:</font></div><div dir="ltr" id="yui_3_16_0_ym19_1_1519085807369_14215"><font id="yui_3_16_0_ym19_1_1519085807369_10517" size="3">Sync Error on Master Domain between Host Node2 and oVirt Engine. Domain Stored2 is marked as master in ovirt engine Database but not on storage side Please consult with support<br id="yui_3_16_0_ym19_1_1519085807369_12607"></font></div><div dir="ltr" id="yui_3_16_0_ym19_1_1519085807369_14204"><font id="yui_3_16_0_ym19_1_1519085807369_10517" size="3"><br></font></div><div dir="ltr" id="yui_3_16_0_ym19_1_1519085807369_15891"><font id="yui_3_16_0_ym19_1_1519085807369_10517" size="3">VDSM Node2 command ConnectStoragePoolVDS failed: Wrong Master domain or it's version: u'SD=f3e372e3-1251-4195-a4b9-1027e40059df, pool=5a865884-0366-0330-02b8-0000</font></div><div dir="ltr" id="yui_3_16_0_ym19_1_1519085807369_15895"><font id="yui_3_16_0_ym19_1_1519085807369_10517" size="3"><br></font></div><div dir="ltr" id="yui_3_16_0_ym19_1_1519085807369_15896"><font id="yui_3_16_0_ym19_1_1519085807369_10517" size="3">VDSM Node2 command HSMGetAllTastsStatusesVDS failed: Not SPM: ()<br id="yui_3_16_0_ym19_1_1519085807369_15901"><br id="yui_3_16_0_ym19_1_1519085807369_15902">Failed to deactivate Storage Domain DATANd01 (Data Center UnsecuredEnv)<br></font></div><div id="yui_3_16_0_ym19_1_1519085807369_10545"><font id="yui_3_16_0_ym19_1_1519085807369_10517" size="3"><br></font></div><div id="yui_3_16_0_ym19_1_1519085807369_10551"><font id="yui_3_16_0_ym19_1_1519085807369_10517" size="3">Here's logs from engine:</font></div><div id="yui_3_16_0_ym19_1_1519085807369_10620"><font id="yui_3_16_0_ym19_1_1519085807369_10517" size="3"><br></font></div><div id="yui_3_16_0_ym19_1_1519085807369_10573"><font id="yui_3_16_0_ym19_1_1519085807369_10517" size="3">-------------------------------------------------------------------------------------------------------</font></div><div id="yui_3_16_0_ym19_1_1519085807369_10588" dir="ltr"><font id="yui_3_16_0_ym19_1_1519085807369_10517" size="3">[root@dev2engine ~]# tail /var/log/messages<br id="yui_3_16_0_ym19_1_1519085807369_10609">Feb 20 07:01:01 dev2engine systemd: Starting Session 20 of user root.<br id="yui_3_16_0_ym19_1_1519085807369_10610">Feb 20 07:01:01 dev2engine systemd: Removed slice User Slice of root.<br id="yui_3_16_0_ym19_1_1519085807369_10611">Feb 20 07:01:01 dev2engine systemd: Stopping User Slice of root.<br id="yui_3_16_0_ym19_1_1519085807369_10612">Feb 20 07:58:52 dev2engine systemd: Created slice User Slice of root.<br id="yui_3_16_0_ym19_1_1519085807369_10613">Feb 20 07:58:52 dev2engine systemd: Starting User Slice of root.<br id="yui_3_16_0_ym19_1_1519085807369_10614">Feb 20 07:58:52 dev2engine systemd-logind: New session 21 of user root.<br id="yui_3_16_0_ym19_1_1519085807369_10615">Feb 20 07:58:52 dev2engine systemd: Started Session 21 of user root.<br id="yui_3_16_0_ym19_1_1519085807369_10616">Feb 20 07:58:52 dev2engine systemd: Starting Session 21 of user root.<br id="yui_3_16_0_ym19_1_1519085807369_10617">Feb 20 08:01:01 dev2engine systemd: Started Session 22 of user root.<br id="yui_3_16_0_ym19_1_1519085807369_10618">Feb 20 08:01:01 dev2engine systemd: Starting Session 22 of user root.</font></div><div dir="ltr" id="yui_3_16_0_ym19_1_1519085807369_10619"><font id="yui_3_16_0_ym19_1_1519085807369_10517" size="3"><br></font></div><div dir="ltr" id="yui_3_16_0_ym19_1_1519085807369_10623"><br><font id="yui_3_16_0_ym19_1_1519085807369_10517" size="3"><font id="yui_3_16_0_ym19_1_1519085807369_10626" size="3">-------------------------------------------------------------------------------------------------------</font></font></div><div dir="ltr" id="yui_3_16_0_ym19_1_1519085807369_10654"><font id="yui_3_16_0_ym19_1_1519085807369_10517" size="3"><font id="yui_3_16_0_ym19_1_1519085807369_10626" size="3">[root@dev2engine ~]# tail /var/log/ovirt-engine/engine.log<br id="yui_3_16_0_ym19_1_1519085807369_10677">2018-02-20 08:01:16,062+08 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-32) [102e9d3c] Finished reconstruct for pool '5a865884-0366-0330-02b8-0000000002d4'. Clearing event queue<br id="yui_3_16_0_ym19_1_1519085807369_10678">2018-02-20 08:01:27,825+08 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-7-thread-23) [] Master domain is not in sync between DB and VDSM. Domain Stored2 marked as master in DB and not in the storage<br id="yui_3_16_0_ym19_1_1519085807369_10679">2018-02-20 08:01:27,862+08 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-23) [213f42b9] Validation of action 'ReconstructMasterDomain' failed for user SYSTEM. Reasons: VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL2,$status PreparingForMaintenance<br id="yui_3_16_0_ym19_1_1519085807369_10680">2018-02-20 08:01:27,882+08 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-20) [929330e] Finished reconstruct for pool '5a865884-0366-0330-02b8-0000000002d4'. Clearing event queue<br id="yui_3_16_0_ym19_1_1519085807369_10681">2018-02-20 08:01:40,106+08 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-7-thread-17) [] Master domain is not in sync between DB and VDSM. Domain Stored2 marked as master in DB and not in the storage<br id="yui_3_16_0_ym19_1_1519085807369_10682">2018-02-20 08:01:40,197+08 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-17) [7af552c1] Validation of action 'ReconstructMasterDomain' failed for user SYSTEM. Reasons: VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL2,$status PreparingForMaintenance<br id="yui_3_16_0_ym19_1_1519085807369_10683">2018-02-20 08:01:40,246+08 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-22) [73673040] Finished reconstruct for pool '5a865884-0366-0330-02b8-0000000002d4'. Clearing event queue<br id="yui_3_16_0_ym19_1_1519085807369_10684">2018-02-20 08:01:51,809+08 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-7-thread-26) [] Master domain is not in sync between DB and VDSM. Domain Stored2 marked as master in DB and not in the storage<br id="yui_3_16_0_ym19_1_1519085807369_10685">2018-02-20 08:01:51,846+08 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-26) [20307cbe] Validation of action 'ReconstructMasterDomain' failed for user SYSTEM. Reasons: VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL2,$status PreparingForMaintenance<br id="yui_3_16_0_ym19_1_1519085807369_10686">2018-02-20 08:01:51,866+08 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-49) [2c11a866] Finished reconstruct for pool '5a865884-0366-0330-02b8-0000000002d4'. Clearing event queue<br id="yui_3_16_0_ym19_1_1519085807369_10687"><br></font></font></div></div></body></html>