<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi, <br>
</p>
<p>We had a failure in OST for test
002_bootstrap.verify_add_all_hosts. <br>
</p>
<p>From the logs I can see that vdsm on host0 was reporting that it
cannot find the physical volume but eventually the storage was
created and is reported as responsive. <br>
</p>
<p>However, Host1 is reported to became non-operational with storage
domain does not exist error and I think that there is a race. <br>
</p>
<p>I think that we create the storage domain while host1 is being
installed and if the domain is not created and reported as
activated in time, host1 will become nonOperational. <br>
</p>
<p>are we starting installation of host1 before host0 and storage
are active? <br>
</p>
<p><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81">
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: I do not think that the patch reported is related to the error</span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><b>
</b></span></p>
</b><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81">
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/84133/">https://gerrit.ovirt.org/#/c/84133/</a><b>
</b></span></p>
<br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job:</span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">
</span></p>
</b><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81">
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3902/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3902/</a>
</span></p>
<br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs:</span></p>
<br>
</b></p>
<p><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3902/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3902/artifact/</a><br>
</b></p>
<p><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: </span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error></span></p>
<br>
Lago log: <br>
</b></p>
<p><span style="font-weight:normal;">2017-11-18
11:15:25,472::log_utils.py::end_log_task::670::nose::INFO:: #
add_master_storage_domain: ESC[32mSuccessESC[0m (in 0:01:09)<br>
2017-11-18
11:15:25,472::log_utils.py::start_log_task::655::nose::INFO:: #
add_secondary_storage_domains: ESC[0mESC[0m<br>
2017-11-18
11:16:47,455::log_utils.py::end_log_task::670::nose::INFO:: #
add_secondary_storage_domains: ESC[32mSuccessESC[0m (in 0:01:21)<br>
2017-11-18
11:16:47,456::log_utils.py::start_log_task::655::nose::INFO:: #
import_templates: ESC[0mESC[0m<br>
2017-11-18
11:16:47,513::testlib.py::stopTest::198::nose::INFO:: *
SKIPPED: Exported domain generation not supported yet<br>
2017-11-18
11:16:47,514::log_utils.py::end_log_task::670::nose::INFO:: #
import_templates: ESC[32mSuccessESC[0m (in 0:00:00)<br>
2017-11-18
11:16:47,514::log_utils.py::start_log_task::655::nose::INFO:: #
verify_add_all_hosts: ESC[0mESC[0m<br>
2017-11-18
11:16:47,719::testlib.py::assert_equals_within::227::ovirtlago.testlib::ERROR::
* Unhandled exception in <function <lambda> at
0x2909230><br>
Traceback (most recent call last):<br>
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py",
line 219, in assert_equals_within<br>
res = func()<br>
File
"/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py",
line 430, in <lambda><br>
lambda: _all_hosts_up(hosts_service, total_hosts)<br>
File
"/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py",
line 129, in _all_hosts_up<br>
_check_problematic_hosts(hosts_service)<br>
File
"/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py",
line 149, in _check_problematic_hosts<br>
raise RuntimeError(dump_hosts)<br>
RuntimeError: 1 hosts failed installation:<br>
lago-basic-suite-master-host-1: non_operational<br>
<br>
2017-11-18
11:16:47,722::utils.py::wrapper::480::lago.utils::DEBUG::Looking
for a workdir<br>
2017-11-18
11:16:47,722::workdir.py::resolve_workdir_path::361::lago.workdir::DEBUG::Checking
if /dev/shm/ost/deployment-basic-suite-master is a workdir<br>
2017-11-18
11:16:47,724::log_utils.py::__enter__::600::lago.prefix::INFO::
* Collect artifacts: ESC[0mESC[0m<br>
2017-11-18
11:16:47,724::log_utils.py::__enter__::600::lago.prefix::INFO::
* Collect artifacts: ESC[0mESC[0m<br>
</span></p>
<p><span style="font-weight:normal;">vdsm host0: <br>
</span></p>
<p><span style="font-weight:normal;">2017-11-18 06:14:23,980-0500
INFO (jsonrpc/0) [vdsm.api] START getDeviceList(storageType=3,
guids=[u'360014059618895272774e97a2aaf5dd6'], checkStatus=False,
options={}) from=::ffff:192.168.201.4,45636,
flow_id=ed8310a1-a7af-4a67-b351-8ff<br>
364766b8a, task_id=6ced0092-34cd-49f0-aa0f-6aae498af37f (api:46)<br>
2017-11-18 06:14:24,353-0500 WARN (jsonrpc/0) [storage.LVM] lvm
pvs failed: 5 [] [' Failed to find physical volume
"/dev/mapper/360014059618895272774e97a2aaf5dd6".'] (lvm:322)<br>
2017-11-18 06:14:24,353-0500 WARN (jsonrpc/0) [storage.HSM]
getPV failed for guid: 360014059618895272774e97a2aaf5dd6
(hsm:1973)<br>
Traceback (most recent call last):<br>
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py",
line 1970, in _getDeviceList<br>
pv = lvm.getPV(guid)<br>
File "/usr/lib/python2.7/site-packages/vdsm/storage/lvm.py",
line 852, in getPV<br>
raise se.InaccessiblePhysDev((pvName,))<br>
InaccessiblePhysDev: Multipath cannot access physical device(s):
"devices=(u'360014059618895272774e97a2aaf5dd6',)"<br>
2017-11-18 06:14:24,389-0500 INFO (jsonrpc/0) [vdsm.api] FINISH
getDeviceList return={'devList': [{'status': 'unknown',
'vendorID': 'LIO-ORG', 'capacity': '21474836480', 'fwrev':
'4.0', 'discard_zeroes_data': 0, 'vgUUID': '', 'pvsize': '',
'pathlist': [{'initiatorname': u'default', 'connection':
u'192.168.200.4', 'iqn': u'iqn.2014-07.org.ovirt:storage',
'portal': '1', 'user': u'username', 'password': '********',
'port': '3260'}, {'initiatorname': u'default', 'connection':
u'192.168.201.4', 'iqn': u'iqn.2014-07.org.ovirt:storage',
'portal': '1', 'user': u'username', 'password': '********',
'port': '3260'}], 'logicalblocksize': '512',
'discard_max_bytes': 1073741824, 'pathstatus': [{'type':
'iSCSI', 'physdev': 'sda', 'capacity': '21474836480', 'state':
'active', 'lun': '0'}, {'type': 'iSCSI', 'physdev': 'sdf',
'capacity': '21474836480', 'state': 'active', 'lun': '0'}],
'devtype': 'iSCSI', 'physicalblocksize': '512', 'pvUUID': '',
'serial':
'SLIO-ORG_lun0_bdev_96188952-7277-4e97-a2aa-f5dd6aad6fc2',
'GUID': '360014059618895272774e97a2aaf5dd6', 'productID':
'lun0_bdev'}]} from=::ffff:192.168.201.4,45636,
flow_id=ed8310a1-a7af-4a67-b351-8ff364766b8a,
task_id=6ced0092-34cd-49f0-aa0f-6aae498af37f (api:52)</span></p>
<p><span style="font-weight:normal;"><br>
</span></p>
<p><span style="font-weight:normal;">2017-11-18 06:14:31,788-0500
INFO (jsonrpc/0) [vdsm.api] FINISH getStorageDomainInfo
return={'info': {'uuid': 'cc61e074-a3b6-4371-9185-66079a39f123',
'vgMetadataDevice': '360014059618895272774e97a2aaf5dd6',
'vguuid': '7ifbmt-0elj-uWZZ-zS<br>
LG-plA8-8hd3-JG298b', 'metadataDevice':
'360014059618895272774e97a2aaf5dd6', 'state': 'OK', 'version':
'4', 'role': 'Regular', 'type': 'ISCSI', 'class': 'Data',
'pool': [], 'name': 'iscsi'}} from=::ffff:192.168.201.4,45636,
flow_id=2c1876<br>
99, task_id=c2080b61-d4a5-4bdb-9d75-f81580a8257a (api:<br>
</span></p>
<p><span style="font-weight:normal;">vdsm host1:</span></p>
<p><span style="font-weight:normal;">2017-11-18 06:16:34,315-0500
ERROR (monitor/c65437c) [storage.Monitor] Setting up monitor for
c65437ce-339f-4b01-aeb5-45c1d486bf49 failed (monitor:329)<br>
Traceback (most recent call last):<br>
File
"/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line
326, in _setupLoop<br>
self._setupMonitor()<br>
File
"/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line
348, in _setupMonitor<br>
self._produceDomain()<br>
File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line
177, in wrapper<br>
value = meth(self, *a, **kw)<br>
File
"/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line
366, in _produceDomain<br>
self.domain = sdCache.produce(self.sdUUID)<br>
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py",
line 110, in produce<br>
domain.getRealDomain()<br>
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py",
line 51, in getRealDomain<br>
return self._cache._realProduce(self._sdUUID)<br>
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py",
line 134, in _realProduce<br>
domain = self._findDomain(sdUUID)<br>
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py",
line 151, in _findDomain<br>
return findMethod(sdUUID)<br>
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py",
line 176, in _findUnfetchedDomain<br>
raise se.StorageDomainDoesNotExist(sdUUID)<br>
StorageDomainDoesNotExist: Storage domain does not exist:
(u'c65437ce-339f-4b01-aeb5-45c1d486bf49',)<br>
2017-11-18 06:16:40,377-0500 INFO (jsonrpc/7) [api.host] START
getStats() from=::ffff:192.168.201.4,58722 (api:46)<br>
2017-11-18 06:16:40,378-0500 INFO (jsonrpc/7) [vdsm.api] START
repoStats(domains=()) from=::ffff:192.168.201.4,58722,
task_id=8fb74944-08c0-491e-ad55-a7a9f0a11ef8 (api:46)<br>
2017-11-18 06:16:40,379-0500 INFO (jsonrpc/7) [vdsm.api] FINISH
repoStats return={u'c65437ce-339f-4b01-aeb5-45c1d486bf49':
{'code': 358, 'actual': True, 'version': -1, 'acquired': False,
'delay': '0', 'lastCheck': '6.1', 'valid': False},<br>
u'cc61e074-a3b6-4371-9185-66079a39f123': {'code': 0, 'actual':
True, 'version': 4, 'acquired': True, 'delay': '0.00103987',
'lastCheck': '6.5', 'valid': True}}
from=::ffff:192.168.201.4,58722,
task_id=8fb74944-08c0-491e-ad55-a7a9f0a11ef8<br>
(api:52)<br>
</span></p>
<p><span style="font-weight:normal;">engine log: <br>
</span></p>
<p><span style="font-weight:normal;">2017-11-18 06:15:54,040-05
ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy]
(EE-ManagedThreadFactory-engine-Thread-29) [4ce8aff3] Domain
'c65437ce-339f-4b01-aeb5-45c1d486bf49:nfs' was reported with
error code '358'<br>
2017-11-18 06:15:54,041-05 ERROR
[org.ovirt.engine.core.bll.InitVdsOnUpCommand]
(EE-ManagedThreadFactory-engine-Thread-29) [4ce8aff3] Storage
Domain 'nfs' of pool 'test-dc' is in problem in host
'lago-basic-suite-master-host-1'<br>
2017-11-18 06:15:54,045-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-29) [4ce8aff3] EVENT_ID:
VDS_STORAGE_VDS_STATS_FAILED(189), Host
lago-basic-suite-master-host-1<br>
reports about one of the Active Storage Domains as Problematic.</span><b
style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><b><br>
</b></b></p>
<p><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><b></b><br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error></span></p>
</b><br class="Apple-interchange-newline">
</p>
<p><br>
</p>
</body>
</html>