<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Nov 20, 2017 at 3:10 PM, Dafna Ron <span dir="ltr"><<a href="mailto:dron@redhat.com" target="_blank">dron@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi, <br>
</p>
<p>We had a failure in OST for test
002_bootstrap.verify_add_all_<wbr>hosts. <br>
</p>
<p>From the logs I can see that vdsm on host0 was reporting that it
cannot find the physical volume but eventually the storage was
created and is reported as responsive. <br>
</p>
<p>However, Host1 is reported to became non-operational with storage
domain does not exist error and I think that there is a race. <br></p></div></blockquote><div><br></div><div>I've opened <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1514906">https://bugzilla.redhat.com/show_bug.cgi?id=1514906</a> on this. </div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><p>
</p>
<p>I think that we create the storage domain while host1 is being
installed and if the domain is not created and reported as
activated in time, host1 will become nonOperational. <br></p></div></blockquote><div><br></div><div>And based on the above description, this is exactly the issue I've described in the BZ.</div><div>Y.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><p>
</p>
<p>are we starting installation of host1 before host0 and storage
are active? <br>
</p>
<p><b style="font-weight:normal" id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81">
</b></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><b style="font-weight:normal" id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Link to suspected patches: I do not think that the patch reported is related to the error</span></b></p><b style="font-weight:normal" id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81">
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><b>
</b></span></p>
</b><b style="font-weight:normal" id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81">
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><a class="gmail-m_-6851820411967884273moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/84133/" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>84133/</a><b>
</b></span></p>
<br>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Link to Job:</span></p>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">
</span></p>
</b><b style="font-weight:normal" id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81">
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><a class="gmail-m_-6851820411967884273moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3902/" target="_blank">http://jenkins.ovirt.org/job/<wbr>ovirt-master_change-queue-<wbr>tester/3902/</a>
</span></p>
<br>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Link to all logs:</span></p>
<br>
</b><p></p>
<p><b style="font-weight:normal" id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><a class="gmail-m_-6851820411967884273moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3902/artifact/" target="_blank">http://jenkins.ovirt.org/job/<wbr>ovirt-master_change-queue-<wbr>tester/3902/artifact/</a><br>
</b></p>
<p><b style="font-weight:normal" id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><br>
</b></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><b style="font-weight:normal" id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">(Relevant) error snippet from the log: </span></b></p><b style="font-weight:normal" id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81">
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><error></span></p>
<br>
Lago log: <br>
</b><p></p>
<p><span style="font-weight:normal">2017-11-18
11:15:25,472::log_utils.py::<wbr>end_log_task::670::nose::INFO:<wbr>: #
add_master_storage_domain: ESC[32mSuccessESC[0m (in 0:01:09)<br>
2017-11-18
11:15:25,472::log_utils.py::<wbr>start_log_task::655::nose::<wbr>INFO:: #
add_secondary_storage_domains: ESC[0mESC[0m<br>
2017-11-18
11:16:47,455::log_utils.py::<wbr>end_log_task::670::nose::INFO:<wbr>: #
add_secondary_storage_domains: ESC[32mSuccessESC[0m (in 0:01:21)<br>
2017-11-18
11:16:47,456::log_utils.py::<wbr>start_log_task::655::nose::<wbr>INFO:: #
import_templates: ESC[0mESC[0m<br>
2017-11-18
11:16:47,513::testlib.py::<wbr>stopTest::198::nose::INFO:: *
SKIPPED: Exported domain generation not supported yet<br>
2017-11-18
11:16:47,514::log_utils.py::<wbr>end_log_task::670::nose::INFO:<wbr>: #
import_templates: ESC[32mSuccessESC[0m (in 0:00:00)<br>
2017-11-18
11:16:47,514::log_utils.py::<wbr>start_log_task::655::nose::<wbr>INFO:: #
verify_add_all_hosts: ESC[0mESC[0m<br>
2017-11-18
11:16:47,719::testlib.py::<wbr>assert_equals_within::227::<wbr>ovirtlago.testlib::ERROR::
* Unhandled exception in <function <lambda> at
0x2909230><br>
Traceback (most recent call last):<br>
File "/usr/lib/python2.7/site-<wbr>packages/ovirtlago/testlib.py"<wbr>,
line 219, in assert_equals_within<br>
res = func()<br>
File
"/home/jenkins/workspace/<wbr>ovirt-master_change-queue-<wbr>tester/ovirt-system-tests/<wbr>basic-suite-master/test-<wbr>scenarios/002_bootstrap.py",
line 430, in <lambda><br>
lambda: _all_hosts_up(hosts_service, total_hosts)<br>
File
"/home/jenkins/workspace/<wbr>ovirt-master_change-queue-<wbr>tester/ovirt-system-tests/<wbr>basic-suite-master/test-<wbr>scenarios/002_bootstrap.py",
line 129, in _all_hosts_up<br>
_check_problematic_hosts(<wbr>hosts_service)<br>
File
"/home/jenkins/workspace/<wbr>ovirt-master_change-queue-<wbr>tester/ovirt-system-tests/<wbr>basic-suite-master/test-<wbr>scenarios/002_bootstrap.py",
line 149, in _check_problematic_hosts<br>
raise RuntimeError(dump_hosts)<br>
RuntimeError: 1 hosts failed installation:<br>
lago-basic-suite-master-host-<wbr>1: non_operational<br>
<br>
2017-11-18
11:16:47,722::utils.py::<wbr>wrapper::480::lago.utils::<wbr>DEBUG::Looking
for a workdir<br>
2017-11-18
11:16:47,722::workdir.py::<wbr>resolve_workdir_path::361::<wbr>lago.workdir::DEBUG::Checking
if /dev/shm/ost/deployment-basic-<wbr>suite-master is a workdir<br>
2017-11-18
11:16:47,724::log_utils.py::__<wbr>enter__::600::lago.prefix::<wbr>INFO::
* Collect artifacts: ESC[0mESC[0m<br>
2017-11-18
11:16:47,724::log_utils.py::__<wbr>enter__::600::lago.prefix::<wbr>INFO::
* Collect artifacts: ESC[0mESC[0m<br>
</span></p>
<p><span style="font-weight:normal">vdsm host0: <br>
</span></p>
<p><span style="font-weight:normal">2017-11-18 06:14:23,980-0500
INFO (jsonrpc/0) [vdsm.api] START getDeviceList(storageType=3,
guids=[u'<wbr>360014059618895272774e97a2aaf5<wbr>dd6'], checkStatus=False,
options={}) from=::ffff:192.168.201.4,<wbr>45636,
flow_id=ed8310a1-a7af-4a67-<wbr>b351-8ff<br>
364766b8a, task_id=6ced0092-34cd-49f0-<wbr>aa0f-6aae498af37f (api:46)<br>
2017-11-18 06:14:24,353-0500 WARN (jsonrpc/0) [storage.LVM] lvm
pvs failed: 5 [] [' Failed to find physical volume
"/dev/mapper/<wbr>360014059618895272774e97a2aaf5<wbr>dd6".'] (lvm:322)<br>
2017-11-18 06:14:24,353-0500 WARN (jsonrpc/0) [storage.HSM]
getPV failed for guid: 360014059618895272774e97a2aaf5<wbr>dd6
(hsm:1973)<br>
Traceback (most recent call last):<br>
File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/hsm.py",
line 1970, in _getDeviceList<br>
pv = lvm.getPV(guid)<br>
File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/lvm.py",
line 852, in getPV<br>
raise se.InaccessiblePhysDev((<wbr>pvName,))<br>
InaccessiblePhysDev: Multipath cannot access physical device(s):
"devices=(u'<wbr>360014059618895272774e97a2aaf5<wbr>dd6',)"<br>
2017-11-18 06:14:24,389-0500 INFO (jsonrpc/0) [vdsm.api] FINISH
getDeviceList return={'devList': [{'status': 'unknown',
'vendorID': 'LIO-ORG', 'capacity': '21474836480', 'fwrev':
'4.0', 'discard_zeroes_data': 0, 'vgUUID': '', 'pvsize': '',
'pathlist': [{'initiatorname': u'default', 'connection':
u'192.168.200.4', 'iqn': u'iqn.2014-07.org.ovirt:<wbr>storage',
'portal': '1', 'user': u'username', 'password': '********',
'port': '3260'}, {'initiatorname': u'default', 'connection':
u'192.168.201.4', 'iqn': u'iqn.2014-07.org.ovirt:<wbr>storage',
'portal': '1', 'user': u'username', 'password': '********',
'port': '3260'}], 'logicalblocksize': '512',
'discard_max_bytes': 1073741824, 'pathstatus': [{'type':
'iSCSI', 'physdev': 'sda', 'capacity': '21474836480', 'state':
'active', 'lun': '0'}, {'type': 'iSCSI', 'physdev': 'sdf',
'capacity': '21474836480', 'state': 'active', 'lun': '0'}],
'devtype': 'iSCSI', 'physicalblocksize': '512', 'pvUUID': '',
'serial':
'SLIO-ORG_lun0_bdev_96188952-<wbr>7277-4e97-a2aa-f5dd6aad6fc2',
'GUID': '<wbr>360014059618895272774e97a2aaf5<wbr>dd6', 'productID':
'lun0_bdev'}]} from=::ffff:192.168.201.4,<wbr>45636,
flow_id=ed8310a1-a7af-4a67-<wbr>b351-8ff364766b8a,
task_id=6ced0092-34cd-49f0-<wbr>aa0f-6aae498af37f (api:52)</span></p>
<p><span style="font-weight:normal"><br>
</span></p>
<p><span style="font-weight:normal">2017-11-18 06:14:31,788-0500
INFO (jsonrpc/0) [vdsm.api] FINISH getStorageDomainInfo
return={'info': {'uuid': 'cc61e074-a3b6-4371-9185-<wbr>66079a39f123',
'vgMetadataDevice': '<wbr>360014059618895272774e97a2aaf5<wbr>dd6',
'vguuid': '7ifbmt-0elj-uWZZ-zS<br>
LG-plA8-8hd3-JG298b', 'metadataDevice':
'<wbr>360014059618895272774e97a2aaf5<wbr>dd6', 'state': 'OK', 'version':
'4', 'role': 'Regular', 'type': 'ISCSI', 'class': 'Data',
'pool': [], 'name': 'iscsi'}} from=::ffff:192.168.201.4,<wbr>45636,
flow_id=2c1876<br>
99, task_id=c2080b61-d4a5-4bdb-<wbr>9d75-f81580a8257a (api:<br>
</span></p>
<p><span style="font-weight:normal">vdsm host1:</span></p>
<p><span style="font-weight:normal">2017-11-18 06:16:34,315-0500
ERROR (monitor/c65437c) [storage.Monitor] Setting up monitor for
c65437ce-339f-4b01-aeb5-<wbr>45c1d486bf49 failed (monitor:329)<br>
Traceback (most recent call last):<br>
File
"/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/monitor.<wbr>py", line
326, in _setupLoop<br>
self._setupMonitor()<br>
File
"/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/monitor.<wbr>py", line
348, in _setupMonitor<br>
self._produceDomain()<br>
File "/usr/lib/python2.7/site-<wbr>packages/vdsm/utils.py", line
177, in wrapper<br>
value = meth(self, *a, **kw)<br>
File
"/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/monitor.<wbr>py", line
366, in _produceDomain<br>
self.domain = sdCache.produce(self.sdUUID)<br>
File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/sdc.py",
line 110, in produce<br>
domain.getRealDomain()<br>
File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/sdc.py",
line 51, in getRealDomain<br>
return self._cache._realProduce(self.<wbr>_sdUUID)<br>
File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/sdc.py",
line 134, in _realProduce<br>
domain = self._findDomain(sdUUID)<br>
File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/sdc.py",
line 151, in _findDomain<br>
return findMethod(sdUUID)<br>
File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/sdc.py",
line 176, in _findUnfetchedDomain<br>
raise se.StorageDomainDoesNotExist(<wbr>sdUUID)<br>
StorageDomainDoesNotExist: Storage domain does not exist:
(u'c65437ce-339f-4b01-aeb5-<wbr>45c1d486bf49',)<br>
2017-11-18 06:16:40,377-0500 INFO (jsonrpc/7) [api.host] START
getStats() from=::ffff:192.168.201.4,<wbr>58722 (api:46)<br>
2017-11-18 06:16:40,378-0500 INFO (jsonrpc/7) [vdsm.api] START
repoStats(domains=()) from=::ffff:192.168.201.4,<wbr>58722,
task_id=8fb74944-08c0-491e-<wbr>ad55-a7a9f0a11ef8 (api:46)<br>
2017-11-18 06:16:40,379-0500 INFO (jsonrpc/7) [vdsm.api] FINISH
repoStats return={u'c65437ce-339f-4b01-<wbr>aeb5-45c1d486bf49':
{'code': 358, 'actual': True, 'version': -1, 'acquired': False,
'delay': '0', 'lastCheck': '6.1', 'valid': False},<br>
u'cc61e074-a3b6-4371-9185-<wbr>66079a39f123': {'code': 0, 'actual':
True, 'version': 4, 'acquired': True, 'delay': '0.00103987',
'lastCheck': '6.5', 'valid': True}}
from=::ffff:192.168.201.4,<wbr>58722,
task_id=8fb74944-08c0-491e-<wbr>ad55-a7a9f0a11ef8<br>
(api:52)<br>
</span></p>
<p><span style="font-weight:normal">engine log: <br>
</span></p>
<p><span style="font-weight:normal">2017-11-18 06:15:54,040-05
ERROR [org.ovirt.engine.core.<wbr>vdsbroker.irsbroker.IrsProxy]
(EE-ManagedThreadFactory-<wbr>engine-Thread-29) [4ce8aff3] Domain
'c65437ce-339f-4b01-aeb5-<wbr>45c1d486bf49:nfs' was reported with
error code '358'<br>
2017-11-18 06:15:54,041-05 ERROR
[org.ovirt.engine.core.bll.<wbr>InitVdsOnUpCommand]
(EE-ManagedThreadFactory-<wbr>engine-Thread-29) [4ce8aff3] Storage
Domain 'nfs' of pool 'test-dc' is in problem in host
'lago-basic-suite-master-host-<wbr>1'<br>
2017-11-18 06:15:54,045-05 ERROR
[org.ovirt.engine.core.dal.<wbr>dbbroker.auditloghandling.<wbr>AuditLogDirector]
(EE-ManagedThreadFactory-<wbr>engine-Thread-29) [4ce8aff3] EVENT_ID:
VDS_STORAGE_VDS_STATS_FAILED(<wbr>189), Host
lago-basic-suite-master-host-1<br>
reports about one of the Active Storage Domains as Problematic.</span><b style="font-weight:normal" id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><b><br>
</b></b></p>
<p><b style="font-weight:normal" id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><b></b><br>
</b></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><b style="font-weight:normal" id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"></error></span></b></p><b style="font-weight:normal" id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81">
</b><br class="gmail-m_-6851820411967884273Apple-interchange-newline">
<p></p>
<p><br>
</p>
</div>
<br>______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br></blockquote></div><br></div></div>