<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 11/20/2017 01:16 PM, Yaniv Kaul
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAJgorsa+XdPcvfknvtJuur3A3r=Zps-orVAC9940RU5=13sNSA@mail.gmail.com">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Nov 20, 2017 at 3:10 PM,
Dafna Ron <span dir="ltr"><<a
href="mailto:dron@redhat.com" target="_blank"
moz-do-not-send="true">dron@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi, <br>
</p>
<p>We had a failure in OST for test
002_bootstrap.verify_add_all_<wbr>hosts. <br>
</p>
<p>From the logs I can see that vdsm on host0 was
reporting that it cannot find the physical volume but
eventually the storage was created and is reported as
responsive. <br>
</p>
<p>However, Host1 is reported to became non-operational
with storage domain does not exist error and I think
that there is a race. <br>
</p>
</div>
</blockquote>
<div><br>
</div>
<div>I've opened <a
href="https://bugzilla.redhat.com/show_bug.cgi?id=1514906"
moz-do-not-send="true">https://bugzilla.redhat.com/show_bug.cgi?id=1514906</a>
on this. </div>
<div> <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p> </p>
<p>I think that we create the storage domain while host1
is being installed and if the domain is not created
and reported as activated in time, host1 will become
nonOperational. <br>
</p>
</div>
</blockquote>
<div><br>
</div>
<div>And based on the above description, this is exactly the
issue I've described in the BZ.</div>
<div>Y.</div>
<div> </div>
</div>
</div>
</div>
</blockquote>
<br>
+1 this seems exactly the same issue. <br>
<br>
<blockquote type="cite"
cite="mid:CAJgorsa+XdPcvfknvtJuur3A3r=Zps-orVAC9940RU5=13sNSA@mail.gmail.com">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p> </p>
<p>are we starting installation of host1 before host0
and storage are active? <br>
</p>
<p><b style="font-weight:normal"
id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81">
</b></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><b
style="font-weight:normal"
id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Link to suspected patches: I do not think that the patch reported is related to the error</span></b></p>
<b style="font-weight:normal"
id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81">
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><b>
</b></span></p>
</b><b style="font-weight:normal"
id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81">
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><a class="gmail-m_-6851820411967884273moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/84133/" target="_blank" moz-do-not-send="true">https://gerrit.ovirt.org/#/c/<wbr>84133/</a><b>
</b></span></p>
<br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Link to Job:</span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">
</span></p>
</b><b style="font-weight:normal"
id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81">
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><a class="gmail-m_-6851820411967884273moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3902/" target="_blank" moz-do-not-send="true">http://jenkins.ovirt.org/job/<wbr>ovirt-master_change-queue-<wbr>tester/3902/</a>
</span></p>
<br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Link to all logs:</span></p>
<br>
</b>
<p><b style="font-weight:normal"
id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><a
class="gmail-m_-6851820411967884273moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3902/artifact/"
target="_blank" moz-do-not-send="true">http://jenkins.ovirt.org/job/<wbr>ovirt-master_change-queue-<wbr>tester/3902/artifact/</a><br>
</b></p>
<p><b style="font-weight:normal"
id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><br>
</b></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><b
style="font-weight:normal"
id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">(Relevant) error snippet from the log: </span></b></p>
<b style="font-weight:normal"
id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81">
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><error></span></p>
<br>
Lago log: <br>
</b>
<p><span style="font-weight:normal">2017-11-18
11:15:25,472::log_utils.py::<wbr>end_log_task::670::nose::INFO:<wbr>:
# add_master_storage_domain: ESC[32mSuccessESC[0m
(in 0:01:09)<br>
2017-11-18 11:15:25,472::log_utils.py::<wbr>start_log_task::655::nose::<wbr>INFO::
# add_secondary_storage_domains: ESC[0mESC[0m<br>
2017-11-18 11:16:47,455::log_utils.py::<wbr>end_log_task::670::nose::INFO:<wbr>:
# add_secondary_storage_domains:
ESC[32mSuccessESC[0m (in 0:01:21)<br>
2017-11-18 11:16:47,456::log_utils.py::<wbr>start_log_task::655::nose::<wbr>INFO::
# import_templates: ESC[0mESC[0m<br>
2017-11-18 11:16:47,513::testlib.py::<wbr>stopTest::198::nose::INFO::
* SKIPPED: Exported domain generation not supported
yet<br>
2017-11-18 11:16:47,514::log_utils.py::<wbr>end_log_task::670::nose::INFO:<wbr>:
# import_templates: ESC[32mSuccessESC[0m (in
0:00:00)<br>
2017-11-18 11:16:47,514::log_utils.py::<wbr>start_log_task::655::nose::<wbr>INFO::
# verify_add_all_hosts: ESC[0mESC[0m<br>
2017-11-18
11:16:47,719::testlib.py::<wbr>assert_equals_within::227::<wbr>ovirtlago.testlib::ERROR::
* Unhandled exception in <function <lambda>
at 0x2909230><br>
Traceback (most recent call last):<br>
File "/usr/lib/python2.7/site-<wbr>packages/ovirtlago/testlib.py"<wbr>,
line 219, in assert_equals_within<br>
res = func()<br>
File
"/home/jenkins/workspace/<wbr>ovirt-master_change-queue-<wbr>tester/ovirt-system-tests/<wbr>basic-suite-master/test-<wbr>scenarios/002_bootstrap.py",
line 430, in <lambda><br>
lambda: _all_hosts_up(hosts_service,
total_hosts)<br>
File
"/home/jenkins/workspace/<wbr>ovirt-master_change-queue-<wbr>tester/ovirt-system-tests/<wbr>basic-suite-master/test-<wbr>scenarios/002_bootstrap.py",
line 129, in _all_hosts_up<br>
_check_problematic_hosts(<wbr>hosts_service)<br>
File
"/home/jenkins/workspace/<wbr>ovirt-master_change-queue-<wbr>tester/ovirt-system-tests/<wbr>basic-suite-master/test-<wbr>scenarios/002_bootstrap.py",
line 149, in _check_problematic_hosts<br>
raise RuntimeError(dump_hosts)<br>
RuntimeError: 1 hosts failed installation:<br>
lago-basic-suite-master-host-<wbr>1: non_operational<br>
<br>
2017-11-18 11:16:47,722::utils.py::<wbr>wrapper::480::lago.utils::<wbr>DEBUG::Looking
for a workdir<br>
2017-11-18
11:16:47,722::workdir.py::<wbr>resolve_workdir_path::361::<wbr>lago.workdir::DEBUG::Checking
if /dev/shm/ost/deployment-basic-<wbr>suite-master
is a workdir<br>
2017-11-18 11:16:47,724::log_utils.py::__<wbr>enter__::600::lago.prefix::<wbr>INFO::
* Collect artifacts: ESC[0mESC[0m<br>
2017-11-18 11:16:47,724::log_utils.py::__<wbr>enter__::600::lago.prefix::<wbr>INFO::
* Collect artifacts: ESC[0mESC[0m<br>
</span></p>
<p><span style="font-weight:normal">vdsm host0: <br>
</span></p>
<p><span style="font-weight:normal">2017-11-18
06:14:23,980-0500 INFO (jsonrpc/0) [vdsm.api] START
getDeviceList(storageType=3, guids=[u'<wbr>360014059618895272774e97a2aaf5<wbr>dd6'],
checkStatus=False, options={})
from=::ffff:192.168.201.4,<wbr>45636,
flow_id=ed8310a1-a7af-4a67-<wbr>b351-8ff<br>
364766b8a, task_id=6ced0092-34cd-49f0-<wbr>aa0f-6aae498af37f
(api:46)<br>
2017-11-18 06:14:24,353-0500 WARN (jsonrpc/0)
[storage.LVM] lvm pvs failed: 5 [] [' Failed to
find physical volume "/dev/mapper/<wbr>360014059618895272774e97a2aaf5<wbr>dd6".']
(lvm:322)<br>
2017-11-18 06:14:24,353-0500 WARN (jsonrpc/0)
[storage.HSM] getPV failed for guid:
360014059618895272774e97a2aaf5<wbr>dd6 (hsm:1973)<br>
Traceback (most recent call last):<br>
File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/hsm.py",
line 1970, in _getDeviceList<br>
pv = lvm.getPV(guid)<br>
File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/lvm.py",
line 852, in getPV<br>
raise se.InaccessiblePhysDev((<wbr>pvName,))<br>
InaccessiblePhysDev: Multipath cannot access
physical device(s): "devices=(u'<wbr>360014059618895272774e97a2aaf5<wbr>dd6',)"<br>
2017-11-18 06:14:24,389-0500 INFO (jsonrpc/0)
[vdsm.api] FINISH getDeviceList return={'devList':
[{'status': 'unknown', 'vendorID': 'LIO-ORG',
'capacity': '21474836480', 'fwrev': '4.0',
'discard_zeroes_data': 0, 'vgUUID': '', 'pvsize':
'', 'pathlist': [{'initiatorname': u'default',
'connection': u'192.168.200.4', 'iqn':
u'iqn.2014-07.org.ovirt:<wbr>storage', 'portal':
'1', 'user': u'username', 'password': '********',
'port': '3260'}, {'initiatorname': u'default',
'connection': u'192.168.201.4', 'iqn':
u'iqn.2014-07.org.ovirt:<wbr>storage', 'portal':
'1', 'user': u'username', 'password': '********',
'port': '3260'}], 'logicalblocksize': '512',
'discard_max_bytes': 1073741824, 'pathstatus':
[{'type': 'iSCSI', 'physdev': 'sda', 'capacity':
'21474836480', 'state': 'active', 'lun': '0'},
{'type': 'iSCSI', 'physdev': 'sdf', 'capacity':
'21474836480', 'state': 'active', 'lun': '0'}],
'devtype': 'iSCSI', 'physicalblocksize': '512',
'pvUUID': '', 'serial':
'SLIO-ORG_lun0_bdev_96188952-<wbr>7277-4e97-a2aa-f5dd6aad6fc2',
'GUID': '<wbr>360014059618895272774e97a2aaf5<wbr>dd6',
'productID': 'lun0_bdev'}]}
from=::ffff:192.168.201.4,<wbr>45636,
flow_id=ed8310a1-a7af-4a67-<wbr>b351-8ff364766b8a,
task_id=6ced0092-34cd-49f0-<wbr>aa0f-6aae498af37f
(api:52)</span></p>
<p><span style="font-weight:normal"><br>
</span></p>
<p><span style="font-weight:normal">2017-11-18
06:14:31,788-0500 INFO (jsonrpc/0) [vdsm.api]
FINISH getStorageDomainInfo return={'info': {'uuid':
'cc61e074-a3b6-4371-9185-<wbr>66079a39f123',
'vgMetadataDevice': '<wbr>360014059618895272774e97a2aaf5<wbr>dd6',
'vguuid': '7ifbmt-0elj-uWZZ-zS<br>
LG-plA8-8hd3-JG298b', 'metadataDevice': '<wbr>360014059618895272774e97a2aaf5<wbr>dd6',
'state': 'OK', 'version': '4', 'role': 'Regular',
'type': 'ISCSI', 'class': 'Data', 'pool': [],
'name': 'iscsi'}} from=::ffff:192.168.201.4,<wbr>45636,
flow_id=2c1876<br>
99, task_id=c2080b61-d4a5-4bdb-<wbr>9d75-f81580a8257a
(api:<br>
</span></p>
<p><span style="font-weight:normal">vdsm host1:</span></p>
<p><span style="font-weight:normal">2017-11-18
06:16:34,315-0500 ERROR (monitor/c65437c)
[storage.Monitor] Setting up monitor for
c65437ce-339f-4b01-aeb5-<wbr>45c1d486bf49 failed
(monitor:329)<br>
Traceback (most recent call last):<br>
File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/monitor.<wbr>py",
line 326, in _setupLoop<br>
self._setupMonitor()<br>
File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/monitor.<wbr>py",
line 348, in _setupMonitor<br>
self._produceDomain()<br>
File "/usr/lib/python2.7/site-<wbr>packages/vdsm/utils.py",
line 177, in wrapper<br>
value = meth(self, *a, **kw)<br>
File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/monitor.<wbr>py",
line 366, in _produceDomain<br>
self.domain = sdCache.produce(self.sdUUID)<br>
File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/sdc.py",
line 110, in produce<br>
domain.getRealDomain()<br>
File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/sdc.py",
line 51, in getRealDomain<br>
return self._cache._realProduce(self.<wbr>_sdUUID)<br>
File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/sdc.py",
line 134, in _realProduce<br>
domain = self._findDomain(sdUUID)<br>
File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/sdc.py",
line 151, in _findDomain<br>
return findMethod(sdUUID)<br>
File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/sdc.py",
line 176, in _findUnfetchedDomain<br>
raise se.StorageDomainDoesNotExist(<wbr>sdUUID)<br>
StorageDomainDoesNotExist: Storage domain does not
exist: (u'c65437ce-339f-4b01-aeb5-<wbr>45c1d486bf49',)<br>
2017-11-18 06:16:40,377-0500 INFO (jsonrpc/7)
[api.host] START getStats()
from=::ffff:192.168.201.4,<wbr>58722 (api:46)<br>
2017-11-18 06:16:40,378-0500 INFO (jsonrpc/7)
[vdsm.api] START repoStats(domains=())
from=::ffff:192.168.201.4,<wbr>58722,
task_id=8fb74944-08c0-491e-<wbr>ad55-a7a9f0a11ef8
(api:46)<br>
2017-11-18 06:16:40,379-0500 INFO (jsonrpc/7)
[vdsm.api] FINISH repoStats
return={u'c65437ce-339f-4b01-<wbr>aeb5-45c1d486bf49':
{'code': 358, 'actual': True, 'version': -1,
'acquired': False, 'delay': '0', 'lastCheck': '6.1',
'valid': False},<br>
u'cc61e074-a3b6-4371-9185-<wbr>66079a39f123':
{'code': 0, 'actual': True, 'version': 4,
'acquired': True, 'delay': '0.00103987',
'lastCheck': '6.5', 'valid': True}}
from=::ffff:192.168.201.4,<wbr>58722,
task_id=8fb74944-08c0-491e-<wbr>ad55-a7a9f0a11ef8<br>
(api:52)<br>
</span></p>
<p><span style="font-weight:normal">engine log: <br>
</span></p>
<p><span style="font-weight:normal">2017-11-18
06:15:54,040-05 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.irsbroker.IrsProxy]
(EE-ManagedThreadFactory-<wbr>engine-Thread-29)
[4ce8aff3] Domain 'c65437ce-339f-4b01-aeb5-<wbr>45c1d486bf49:nfs'
was reported with error code '358'<br>
2017-11-18 06:15:54,041-05 ERROR
[org.ovirt.engine.core.bll.<wbr>InitVdsOnUpCommand]
(EE-ManagedThreadFactory-<wbr>engine-Thread-29)
[4ce8aff3] Storage Domain 'nfs' of pool 'test-dc' is
in problem in host 'lago-basic-suite-master-host-<wbr>1'<br>
2017-11-18 06:15:54,045-05 ERROR
[org.ovirt.engine.core.dal.<wbr>dbbroker.auditloghandling.<wbr>AuditLogDirector]
(EE-ManagedThreadFactory-<wbr>engine-Thread-29)
[4ce8aff3] EVENT_ID: VDS_STORAGE_VDS_STATS_FAILED(<wbr>189),
Host lago-basic-suite-master-host-1<br>
reports about one of the Active Storage Domains as
Problematic.</span><b style="font-weight:normal"
id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><b><br>
</b></b></p>
<p><b style="font-weight:normal"
id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><br>
</b></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><b
style="font-weight:normal"
id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"></error></span></b></p>
<b style="font-weight:normal"
id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81">
</b><br
class="gmail-m_-6851820411967884273Apple-interchange-newline">
<p><br>
</p>
</div>
<br>
______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org" moz-do-not-send="true">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<p><br>
</p>
</body>
</html>