<div dir="ltr"><div><div><div><div><div>Hi, <br><br></div>We failed cq test 002_bootstrap.verify_add_all_hosts for Master vdsm project. <br><br></div>Looking at the log, vdsm cannot find master storage domain and engine puts the host on non-operational state. <br><br></div>Although on the surface the patch seems to be related, the master storage domain is iscsi whole the patch is related to gluster. <br><br></div>I do not think there is a connection between the patch and the failure but can you please have a look to make sure? <br><br><b style="font-weight:normal" id="gmail-docs-internal-guid-5859b7a1-51ab-0d5c-22c2-c567891e775f"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Link and headline of suspected patches: </span></p><a href="https://gerrit.ovirt.org/#/c/69668/">https://gerrit.ovirt.org/#/c/69668/</a> - </b><br><b style="font-weight:normal" id="gmail-docs-internal-guid-5859b7a1-51ab-0d5c-22c2-c567891e775f">gluster: Fix error when brick is on a btrfs subvolume<br><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Link to Job:</span></p><a href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5180/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5180/</a><br><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Link to all logs:</span></p><a href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5180/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5180/artifact/</a><br><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">(Relevant) error snippet from the log: </span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">&lt;error&gt;</span></p><br><br><b>vdsm: <br><br></b>2018-02-01 03:13:49,211-0500 INFO  (jsonrpc/4) [vdsm.api] START createStorageDomain(storageType=3, sdUUID=u&#39;077add35-9171-45d5-b6de-79cc5a853c36&#39;, domainName=u&#39;iscsi&#39;, typeSpecificArg=u&#39;IdW3HG-K1Af-e0d3-u2O3-rGle-8fk5-ACNk6C&#39;, domClass=1,<br> domVersion=u&#39;4&#39;, options=None) from=::ffff:192.168.201.4,58530, flow_id=22d4ffd8, task_id=2ce6dd52-3d28-4532-abbf-d78d52af6cda (api:46)<br><br>2018-02-01 03:14:40,223-0500 INFO  (jsonrpc/7) [vdsm.api] START connectStoragePool(spUUID=u&#39;2570c0c9-f872-4e49-964a-ee533a79c3f2&#39;, hostID=1, msdUUID=u&#39;077add35-9171-45d5-b6de-79cc5a853c36&#39;, masterVersion=1, domainsMap={u&#39;077add35-9171-45d<br>5-b6de-79cc5a853c36&#39;: u&#39;active&#39;}, options=None) from=::ffff:192.168.201.4,36310, flow_id=19e9aa89, task_id=878419a0-c5ce-4e35-aed5-b27d56b2886e (api:46)<br>2018-02-01 03:14:40,225-0500 INFO  (jsonrpc/7) [storage.StoragePoolMemoryBackend] new storage pool master version 1 and domains map {u&#39;077add35-9171-45d5-b6de-79cc5a853c36&#39;: u&#39;Active&#39;} (spbackends:449)<br>2018-02-01 03:14:40,225-0500 INFO  (jsonrpc/7) [storage.StoragePool] updating pool 2570c0c9-f872-4e49-964a-ee533a79c3f2 backend from type NoneType instance 0x7f45919e3f20 to type StoragePoolMemoryBackend instance 0x45411b0 (sp:157)<br>2018-02-01 03:14:40,226-0500 INFO  (jsonrpc/7) [storage.StoragePool] Connect host #1 to the storage pool 2570c0c9-f872-4e49-964a-ee533a79c3f2 with master domain: 077add35-9171-45d5-b6de-79cc5a853c36 (ver = 1) (sp:692)<br>2018-02-01 03:14:40,462-0500 INFO  (jsonrpc/7) [vdsm.api] FINISH connectStoragePool error=Cannot find master domain: u&#39;spUUID=2570c0c9-f872-4e49-964a-ee533a79c3f2, msdUUID=077add35-9171-45d5-b6de-79cc5a853c36&#39; from=::ffff:192.168.201.4,36<br>310, flow_id=19e9aa89, task_id=878419a0-c5ce-4e35-aed5-b27d56b2886e (api:50)<br>2018-02-01 03:14:40,462-0500 ERROR (jsonrpc/7) [storage.TaskManager.Task] (Task=&#39;878419a0-c5ce-4e35-aed5-b27d56b2886e&#39;) Unexpected error (task:875)<br>Traceback (most recent call last):<br>  File &quot;/usr/lib/python2.7/site-packages/vdsm/storage/task.py&quot;, line 882, in _run<br>    return fn(*args, **kargs)<br>  File &quot;&lt;string&gt;&quot;, line 2, in connectStoragePool<br>  File &quot;/usr/lib/python2.7/site-packages/vdsm/common/api.py&quot;, line 48, in method<br>    ret = func(*args, **kwargs)<br>  File &quot;/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py&quot;, line 1032, in connectStoragePool<br>    spUUID, hostID, msdUUID, masterVersion, domainsMap)<br>  File &quot;/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py&quot;, line 1094, in _connectStoragePool<br>    res = pool.connect(hostID, msdUUID, masterVersion)<br>  File &quot;/usr/lib/python2.7/site-packages/vdsm/storage/sp.py&quot;, line 704, in connect<br>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)<br>  File &quot;/usr/lib/python2.7/site-packages/vdsm/storage/sp.py&quot;, line 1275, in __rebuild<br>    self.setMasterDomain(msdUUID, masterVersion)<br>  File &quot;/usr/lib/python2.7/site-packages/vdsm/storage/sp.py&quot;, line 1488, in setMasterDomain<br>    raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)<br>StoragePoolMasterNotFound: Cannot find master domain: u&#39;spUUID=2570c0c9-f872-4e49-964a-ee533a79c3f2, msdUUID=077add35-9171-45d5-b6de-79cc5a853c36&#39;<br>2018-02-01 03:14:40,466-0500 INFO  (jsonrpc/7) [storage.TaskManager.Task] (Task=&#39;878419a0-c5ce-4e35-aed5-b27d56b2886e&#39;) aborting: Task is aborted: &quot;Cannot find master domain: u&#39;spUUID=2570c0c9-f872-4e49-964a-ee533a79c3f2, msdUUID=077add35<br>-9171-45d5-b6de-79cc5a853c36&#39;&quot; - code 304 (task:1181)<br>2018-02-01 03:14:40,467-0500 ERROR (jsonrpc/7) [storage.Dispatcher] FINISH connectStoragePool error=Cannot find master domain: u&#39;spUUID=2570c0c9-f872-4e49-964a-ee533a79c3f2, msdUUID=077add35-9171-45d5-b6de-79cc5a853c36&#39; (dispatcher:82)<br>2018-02-01 03:14:40,467-0500 INFO  (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.connect failed (error 304) in 0.25 seconds (__init__:573)<br><br></b></div><b style="font-weight:normal" id="gmail-docs-internal-guid-5859b7a1-51ab-0d5c-22c2-c567891e775f"><b>engine:<br></b></b><span style="font-weight:normal"><br>2018-02-01 03:14:40,603-05 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [ba52086] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Host lago-basic-suite-mast<br>er-host-0 cannot access the Storage Domain(s) &lt;UNKNOWN&gt; attached to the Data Center test-dc. Setting Host state to Non-Operational.<br>2018-02-01 03:14:40,608-05 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [ba52086] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Failed to verify Pow<br>er Management configuration for Host lago-basic-suite-master-host-0.<br>2018-02-01 03:14:40,610-05 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [ba52086] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Failed to connect Host lago-<br>basic-suite-master-host-0 to Storage Pool test-dc<br></span><b style="font-weight:normal" id="gmail-docs-internal-guid-5859b7a1-51ab-0d5c-22c2-c567891e775f"><b><br></b></b><div><div><b style="font-weight:normal" id="gmail-docs-internal-guid-5859b7a1-51ab-0d5c-22c2-c567891e775f"><br><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">&lt;/error&gt;</span></p></b><br class="gmail-Apple-interchange-newline"><div><div><br><br></div></div></div></div></div>