<div dir="ltr">Hi, <div><br></div><div>Can you please attach full Engine and VDSM logs?</div><div><br></div><div>Thanks,</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Feb 20, 2018 at 2:59 AM, michael pagdanganan <span dir="ltr"><<a href="mailto:mhke_aj5566@yahoo.com" target="_blank">mhke_aj5566@yahoo.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="color:#000;background-color:#fff;font-family:Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif;font-size:10px"><div id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10542" dir="ltr"><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_15967" size="3">My storage pool has 2 Master Domain(</font><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_16063" size="3"><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_15968" size="3">Stored2 on Node2,Node1Container on Node1)</font>. and my old master domain </font><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_15934" size="3"><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_16064" size="3">(</font><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_16065" size="3"><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_16066" size="3">DATANd01 on Node1 )</font></font> hung on preparing for maintenance. When I tried to activate my old master domain (</font><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3"><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_15935" size="3">DATANd01 on Node1 )</font> all storage domain goes down and up master keep on rotating.</font></div><div id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10905"><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3"><br></font></div><div id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10692"><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3"><br></font></div><div id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10710" dir="ltr"><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3">Ovirt Version (oVirt Engine Version: 4.1.9.1.el7.centos)<br></font></div><div dir="ltr" id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_14202"><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3"><br></font></div><div dir="ltr" id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_12604"><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3"><br></font></div><div dir="ltr" id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_14203"><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3">Event Error:</font></div><div dir="ltr" id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_14215"><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3">Sync Error on Master Domain between Host Node2 and oVirt Engine. Domain Stored2 is marked as master in ovirt engine Database but not on storage side Please consult with support<br id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_12607"></font></div><div dir="ltr" id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_14204"><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3"><br></font></div><div dir="ltr" id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_15891"><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3">VDSM Node2 command ConnectStoragePoolVDS failed: Wrong Master domain or it's version: u'SD=f3e372e3-1251-4195-a4b9-<wbr>1027e40059df, pool=5a865884-0366-0330-02b8-<wbr>0000</font></div><div dir="ltr" id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_15895"><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3"><br></font></div><div dir="ltr" id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_15896"><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3">VDSM Node2 command HSMGetAllTastsStatusesVDS failed: Not SPM: ()<br id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_15901"><br id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_15902">Failed to deactivate Storage Domain DATANd01 (Data Center UnsecuredEnv)<br></font></div><div id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10545"><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3"><br></font></div><div id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10551"><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3">Here's logs from engine:</font></div><div id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10620"><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3"><br></font></div><div id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10573"><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3">------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-------------</font></div><div id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10588" dir="ltr"><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3">[root@dev2engine ~]# tail /var/log/messages<br id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10609">Feb 20 07:01:01 dev2engine systemd: Starting Session 20 of user root.<br id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10610">Feb 20 07:01:01 dev2engine systemd: Removed slice User Slice of root.<br id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10611">Feb 20 07:01:01 dev2engine systemd: Stopping User Slice of root.<br id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10612">Feb 20 07:58:52 dev2engine systemd: Created slice User Slice of root.<br id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10613">Feb 20 07:58:52 dev2engine systemd: Starting User Slice of root.<br id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10614">Feb 20 07:58:52 dev2engine systemd-logind: New session 21 of user root.<br id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10615">Feb 20 07:58:52 dev2engine systemd: Started Session 21 of user root.<br id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10616">Feb 20 07:58:52 dev2engine systemd: Starting Session 21 of user root.<br id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10617">Feb 20 08:01:01 dev2engine systemd: Started Session 22 of user root.<br id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10618">Feb 20 08:01:01 dev2engine systemd: Starting Session 22 of user root.</font></div><div dir="ltr" id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10619"><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3"><br></font></div><div dir="ltr" id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10623"><br><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3"><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10626" size="3">------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-------------</font></font></div><div dir="ltr" id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10654"><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10517" size="3"><font id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10626" size="3">[root@dev2engine ~]# tail /var/log/ovirt-engine/engine.<wbr>log<br id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10677">2018-02-20 08:01:16,062+08 INFO [org.ovirt.engine.core.bll.<wbr>eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-<wbr>thread-32) [102e9d3c] Finished reconstruct for pool '5a865884-0366-0330-02b8-<wbr>0000000002d4'. Clearing event queue<br id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10678">2018-02-20 08:01:27,825+08 WARN [org.ovirt.engine.core.<wbr>vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-7-<wbr>thread-23) [] Master domain is not in sync between DB and VDSM. Domain Stored2 marked as master in DB and not in the storage<br id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10679">2018-02-20 08:01:27,862+08 WARN [org.ovirt.engine.core.bll.<wbr>storage.pool.<wbr>ReconstructMasterDomainCommand<wbr>] (org.ovirt.thread.pool-7-<wbr>thread-23) [213f42b9] Validation of action 'ReconstructMasterDomain' failed for user SYSTEM. Reasons: VAR__ACTION__RECONSTRUCT_<wbr>MASTER,VAR__TYPE__STORAGE__<wbr>DOMAIN,ACTION_TYPE_FAILED_<wbr>STORAGE_DOMAIN_STATUS_<wbr>ILLEGAL2,$status PreparingForMaintenance<br id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10680">2018-02-20 08:01:27,882+08 INFO [org.ovirt.engine.core.bll.<wbr>eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-<wbr>thread-20) [929330e] Finished reconstruct for pool '5a865884-0366-0330-02b8-<wbr>0000000002d4'. Clearing event queue<br id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10681">2018-02-20 08:01:40,106+08 WARN [org.ovirt.engine.core.<wbr>vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-7-<wbr>thread-17) [] Master domain is not in sync between DB and VDSM. Domain Stored2 marked as master in DB and not in the storage<br id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10682">2018-02-20 08:01:40,197+08 WARN [org.ovirt.engine.core.bll.<wbr>storage.pool.<wbr>ReconstructMasterDomainCommand<wbr>] (org.ovirt.thread.pool-7-<wbr>thread-17) [7af552c1] Validation of action 'ReconstructMasterDomain' failed for user SYSTEM. Reasons: VAR__ACTION__RECONSTRUCT_<wbr>MASTER,VAR__TYPE__STORAGE__<wbr>DOMAIN,ACTION_TYPE_FAILED_<wbr>STORAGE_DOMAIN_STATUS_<wbr>ILLEGAL2,$status PreparingForMaintenance<br id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10683">2018-02-20 08:01:40,246+08 INFO [org.ovirt.engine.core.bll.<wbr>eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-<wbr>thread-22) [73673040] Finished reconstruct for pool '5a865884-0366-0330-02b8-<wbr>0000000002d4'. Clearing event queue<br id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10684">2018-02-20 08:01:51,809+08 WARN [org.ovirt.engine.core.<wbr>vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-7-<wbr>thread-26) [] Master domain is not in sync between DB and VDSM. Domain Stored2 marked as master in DB and not in the storage<br id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10685">2018-02-20 08:01:51,846+08 WARN [org.ovirt.engine.core.bll.<wbr>storage.pool.<wbr>ReconstructMasterDomainCommand<wbr>] (org.ovirt.thread.pool-7-<wbr>thread-26) [20307cbe] Validation of action 'ReconstructMasterDomain' failed for user SYSTEM. Reasons: VAR__ACTION__RECONSTRUCT_<wbr>MASTER,VAR__TYPE__STORAGE__<wbr>DOMAIN,ACTION_TYPE_FAILED_<wbr>STORAGE_DOMAIN_STATUS_<wbr>ILLEGAL2,$status PreparingForMaintenance<br id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10686">2018-02-20 08:01:51,866+08 INFO [org.ovirt.engine.core.bll.<wbr>eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-<wbr>thread-49) [2c11a866] Finished reconstruct for pool '5a865884-0366-0330-02b8-<wbr>0000000002d4'. Clearing event queue<br id="m_-4605301492274769828yui_3_16_0_ym19_1_1519085807369_10687"><br></font></font></div></div></div><br>______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Regards,<div>Eyal Shenitzky</div></div></div>
</div>