<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <div class="moz-cite-prefix">On 11/20/2017 01:16 PM, Yaniv Kaul
      wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CAJgorsa+XdPcvfknvtJuur3A3r=Zps-orVAC9940RU5=13sNSA@mail.gmail.com">
      <div dir="ltr"><br>
        <div class="gmail_extra"><br>
          <div class="gmail_quote">On Mon, Nov 20, 2017 at 3:10 PM,
            Dafna Ron <span dir="ltr">&lt;<a
                href="mailto:dron@redhat.com" target="_blank"
                moz-do-not-send="true">dron@redhat.com</a>&gt;</span>
            wrote:<br>
            <blockquote class="gmail_quote" style="margin:0px 0px 0px
              0.8ex;border-left:1px solid
              rgb(204,204,204);padding-left:1ex">
              <div bgcolor="#FFFFFF">
                <p>Hi, <br>
                </p>
                <p>We had a failure in OST for test
                  002_bootstrap.verify_add_all_<wbr>hosts. <br>
                </p>
                <p>From the logs I can see that vdsm on host0 was
                  reporting that it cannot find the physical volume but
                  eventually the storage was created and is reported as
                  responsive. <br>
                </p>
                <p>However, Host1 is reported to became non-operational
                  with storage domain does not exist error and I think
                  that there is a race. <br>
                </p>
              </div>
            </blockquote>
            <div><br>
            </div>
            <div>I've opened <a
                href="https://bugzilla.redhat.com/show_bug.cgi?id=1514906"
                moz-do-not-send="true">https://bugzilla.redhat.com/show_bug.cgi?id=1514906</a>
              on this. </div>
            <div> <br>
            </div>
            <blockquote class="gmail_quote" style="margin:0px 0px 0px
              0.8ex;border-left:1px solid
              rgb(204,204,204);padding-left:1ex">
              <div bgcolor="#FFFFFF">
                <p> </p>
                <p>I think that we create the storage domain while host1
                  is being installed and if the domain is not created
                  and reported as activated in time, host1 will become
                  nonOperational. <br>
                </p>
              </div>
            </blockquote>
            <div><br>
            </div>
            <div>And based on the above description, this is exactly the
              issue I've described in the BZ.</div>
            <div>Y.</div>
            <div> </div>
          </div>
        </div>
      </div>
    </blockquote>
    <br>
    +1 this seems exactly the same issue. <br>
    <br>
    <blockquote type="cite"
cite="mid:CAJgorsa+XdPcvfknvtJuur3A3r=Zps-orVAC9940RU5=13sNSA@mail.gmail.com">
      <div dir="ltr">
        <div class="gmail_extra">
          <div class="gmail_quote">
            <blockquote class="gmail_quote" style="margin:0px 0px 0px
              0.8ex;border-left:1px solid
              rgb(204,204,204);padding-left:1ex">
              <div bgcolor="#FFFFFF">
                <p> </p>
                <p>are we starting installation of host1 before host0
                  and storage are active? <br>
                </p>
                <p><b style="font-weight:normal"
id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81">
                  </b></p>
                <p dir="ltr"
                  style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><b
                    style="font-weight:normal"
id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Link to suspected patches: I do not think that the patch reported is related to the error</span></b></p>
                <b style="font-weight:normal"
id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81">
                  <p dir="ltr"
                    style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><b>
</b></span></p>
                </b><b style="font-weight:normal"
id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81">
                  <p dir="ltr"
                    style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><a class="gmail-m_-6851820411967884273moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/84133/" target="_blank" moz-do-not-send="true">https://gerrit.ovirt.org/#/c/<wbr>84133/</a><b>
</b></span></p>
                  <br>
                  <p dir="ltr"
                    style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Link to Job:</span></p>
                  <p dir="ltr"
                    style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">
</span></p>
                </b><b style="font-weight:normal"
id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81">
                  <p dir="ltr"
                    style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><a class="gmail-m_-6851820411967884273moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3902/" target="_blank" moz-do-not-send="true">http://jenkins.ovirt.org/job/<wbr>ovirt-master_change-queue-<wbr>tester/3902/</a>
</span></p>
                  <br>
                  <p dir="ltr"
                    style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Link to all logs:</span></p>
                  <br>
                </b>
                <p><b style="font-weight:normal"
id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><a
class="gmail-m_-6851820411967884273moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3902/artifact/"
                      target="_blank" moz-do-not-send="true">http://jenkins.ovirt.org/job/<wbr>ovirt-master_change-queue-<wbr>tester/3902/artifact/</a><br>
                  </b></p>
                <p><b style="font-weight:normal"
id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><br>
                  </b></p>
                <p dir="ltr"
                  style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><b
                    style="font-weight:normal"
id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">(Relevant) error snippet from the log: </span></b></p>
                <b style="font-weight:normal"
id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81">
                  <p dir="ltr"
                    style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">&lt;error&gt;</span></p>
                  <br>
                  Lago log: <br>
                </b>
                <p><span style="font-weight:normal">2017-11-18
                    11:15:25,472::log_utils.py::<wbr>end_log_task::670::nose::INFO:<wbr>: 
                    # add_master_storage_domain: ESC[32mSuccessESC[0m
                    (in 0:01:09)<br>
                    2017-11-18 11:15:25,472::log_utils.py::<wbr>start_log_task::655::nose::<wbr>INFO:: 
                    # add_secondary_storage_domains: ESC[0mESC[0m<br>
                    2017-11-18 11:16:47,455::log_utils.py::<wbr>end_log_task::670::nose::INFO:<wbr>: 
                    # add_secondary_storage_domains:
                    ESC[32mSuccessESC[0m (in 0:01:21)<br>
                    2017-11-18 11:16:47,456::log_utils.py::<wbr>start_log_task::655::nose::<wbr>INFO:: 
                    # import_templates: ESC[0mESC[0m<br>
                    2017-11-18 11:16:47,513::testlib.py::<wbr>stopTest::198::nose::INFO::   
                    * SKIPPED: Exported domain generation not supported
                    yet<br>
                    2017-11-18 11:16:47,514::log_utils.py::<wbr>end_log_task::670::nose::INFO:<wbr>: 
                    # import_templates: ESC[32mSuccessESC[0m (in
                    0:00:00)<br>
                    2017-11-18 11:16:47,514::log_utils.py::<wbr>start_log_task::655::nose::<wbr>INFO:: 
                    # verify_add_all_hosts: ESC[0mESC[0m<br>
                    2017-11-18
                    11:16:47,719::testlib.py::<wbr>assert_equals_within::227::<wbr>ovirtlago.testlib::ERROR::   
                    * Unhandled exception in &lt;function &lt;lambda&gt;
                    at 0x2909230&gt;<br>
                    Traceback (most recent call last):<br>
                      File "/usr/lib/python2.7/site-<wbr>packages/ovirtlago/testlib.py"<wbr>,
                    line 219, in assert_equals_within<br>
                        res = func()<br>
                      File
                    "/home/jenkins/workspace/<wbr>ovirt-master_change-queue-<wbr>tester/ovirt-system-tests/<wbr>basic-suite-master/test-<wbr>scenarios/002_bootstrap.py",
                    line 430, in &lt;lambda&gt;<br>
                        lambda: _all_hosts_up(hosts_service,
                    total_hosts)<br>
                      File
                    "/home/jenkins/workspace/<wbr>ovirt-master_change-queue-<wbr>tester/ovirt-system-tests/<wbr>basic-suite-master/test-<wbr>scenarios/002_bootstrap.py",
                    line 129, in _all_hosts_up<br>
                        _check_problematic_hosts(<wbr>hosts_service)<br>
                      File
                    "/home/jenkins/workspace/<wbr>ovirt-master_change-queue-<wbr>tester/ovirt-system-tests/<wbr>basic-suite-master/test-<wbr>scenarios/002_bootstrap.py",
                    line 149, in _check_problematic_hosts<br>
                        raise RuntimeError(dump_hosts)<br>
                    RuntimeError: 1 hosts failed installation:<br>
                    lago-basic-suite-master-host-<wbr>1: non_operational<br>
                    <br>
                    2017-11-18 11:16:47,722::utils.py::<wbr>wrapper::480::lago.utils::<wbr>DEBUG::Looking
                    for a workdir<br>
                    2017-11-18
                    11:16:47,722::workdir.py::<wbr>resolve_workdir_path::361::<wbr>lago.workdir::DEBUG::Checking
                    if /dev/shm/ost/deployment-basic-<wbr>suite-master
                    is a workdir<br>
                    2017-11-18 11:16:47,724::log_utils.py::__<wbr>enter__::600::lago.prefix::<wbr>INFO::   
                    * Collect artifacts: ESC[0mESC[0m<br>
                    2017-11-18 11:16:47,724::log_utils.py::__<wbr>enter__::600::lago.prefix::<wbr>INFO::   
                    * Collect artifacts: ESC[0mESC[0m<br>
                  </span></p>
                <p><span style="font-weight:normal">vdsm host0: <br>
                  </span></p>
                <p><span style="font-weight:normal">2017-11-18
                    06:14:23,980-0500 INFO  (jsonrpc/0) [vdsm.api] START
                    getDeviceList(storageType=3, guids=[u'<wbr>360014059618895272774e97a2aaf5<wbr>dd6'],
                    checkStatus=False, options={})
                    from=::ffff:192.168.201.4,<wbr>45636,
                    flow_id=ed8310a1-a7af-4a67-<wbr>b351-8ff<br>
                    364766b8a, task_id=6ced0092-34cd-49f0-<wbr>aa0f-6aae498af37f
                    (api:46)<br>
                    2017-11-18 06:14:24,353-0500 WARN  (jsonrpc/0)
                    [storage.LVM] lvm pvs failed: 5 [] ['  Failed to
                    find physical volume "/dev/mapper/<wbr>360014059618895272774e97a2aaf5<wbr>dd6".']
                    (lvm:322)<br>
                    2017-11-18 06:14:24,353-0500 WARN  (jsonrpc/0)
                    [storage.HSM] getPV failed for guid:
                    360014059618895272774e97a2aaf5<wbr>dd6 (hsm:1973)<br>
                    Traceback (most recent call last):<br>
                      File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/hsm.py",
                    line 1970, in _getDeviceList<br>
                        pv = lvm.getPV(guid)<br>
                      File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/lvm.py",
                    line 852, in getPV<br>
                        raise se.InaccessiblePhysDev((<wbr>pvName,))<br>
                    InaccessiblePhysDev: Multipath cannot access
                    physical device(s): "devices=(u'<wbr>360014059618895272774e97a2aaf5<wbr>dd6',)"<br>
                    2017-11-18 06:14:24,389-0500 INFO  (jsonrpc/0)
                    [vdsm.api] FINISH getDeviceList return={'devList':
                    [{'status': 'unknown', 'vendorID': 'LIO-ORG',
                    'capacity': '21474836480', 'fwrev': '4.0',
                    'discard_zeroes_data': 0, 'vgUUID': '', 'pvsize':
                    '', 'pathlist': [{'initiatorname': u'default',
                    'connection': u'192.168.200.4', 'iqn':
                    u'iqn.2014-07.org.ovirt:<wbr>storage', 'portal':
                    '1', 'user': u'username', 'password': '********',
                    'port': '3260'}, {'initiatorname': u'default',
                    'connection': u'192.168.201.4', 'iqn':
                    u'iqn.2014-07.org.ovirt:<wbr>storage', 'portal':
                    '1', 'user': u'username', 'password': '********',
                    'port': '3260'}], 'logicalblocksize': '512',
                    'discard_max_bytes': 1073741824, 'pathstatus':
                    [{'type': 'iSCSI', 'physdev': 'sda', 'capacity':
                    '21474836480', 'state': 'active', 'lun': '0'},
                    {'type': 'iSCSI', 'physdev': 'sdf', 'capacity':
                    '21474836480', 'state': 'active', 'lun': '0'}],
                    'devtype': 'iSCSI', 'physicalblocksize': '512',
                    'pvUUID': '', 'serial':
                    'SLIO-ORG_lun0_bdev_96188952-<wbr>7277-4e97-a2aa-f5dd6aad6fc2',
                    'GUID': '<wbr>360014059618895272774e97a2aaf5<wbr>dd6',
                    'productID': 'lun0_bdev'}]}
                    from=::ffff:192.168.201.4,<wbr>45636,
                    flow_id=ed8310a1-a7af-4a67-<wbr>b351-8ff364766b8a,
                    task_id=6ced0092-34cd-49f0-<wbr>aa0f-6aae498af37f
                    (api:52)</span></p>
                <p><span style="font-weight:normal"><br>
                  </span></p>
                <p><span style="font-weight:normal">2017-11-18
                    06:14:31,788-0500 INFO  (jsonrpc/0) [vdsm.api]
                    FINISH getStorageDomainInfo return={'info': {'uuid':
                    'cc61e074-a3b6-4371-9185-<wbr>66079a39f123',
                    'vgMetadataDevice': '<wbr>360014059618895272774e97a2aaf5<wbr>dd6',
                    'vguuid': '7ifbmt-0elj-uWZZ-zS<br>
                    LG-plA8-8hd3-JG298b', 'metadataDevice': '<wbr>360014059618895272774e97a2aaf5<wbr>dd6',
                    'state': 'OK', 'version': '4', 'role': 'Regular',
                    'type': 'ISCSI', 'class': 'Data', 'pool': [],
                    'name': 'iscsi'}} from=::ffff:192.168.201.4,<wbr>45636,
                    flow_id=2c1876<br>
                    99, task_id=c2080b61-d4a5-4bdb-<wbr>9d75-f81580a8257a
                    (api:<br>
                  </span></p>
                <p><span style="font-weight:normal">vdsm host1:</span></p>
                <p><span style="font-weight:normal">2017-11-18
                    06:16:34,315-0500 ERROR (monitor/c65437c)
                    [storage.Monitor] Setting up monitor for
                    c65437ce-339f-4b01-aeb5-<wbr>45c1d486bf49 failed
                    (monitor:329)<br>
                    Traceback (most recent call last):<br>
                      File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/monitor.<wbr>py",
                    line 326, in _setupLoop<br>
                        self._setupMonitor()<br>
                      File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/monitor.<wbr>py",
                    line 348, in _setupMonitor<br>
                        self._produceDomain()<br>
                      File "/usr/lib/python2.7/site-<wbr>packages/vdsm/utils.py",
                    line 177, in wrapper<br>
                        value = meth(self, *a, **kw)<br>
                      File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/monitor.<wbr>py",
                    line 366, in _produceDomain<br>
                        self.domain = sdCache.produce(self.sdUUID)<br>
                      File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/sdc.py",
                    line 110, in produce<br>
                        domain.getRealDomain()<br>
                      File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/sdc.py",
                    line 51, in getRealDomain<br>
                        return self._cache._realProduce(self.<wbr>_sdUUID)<br>
                      File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/sdc.py",
                    line 134, in _realProduce<br>
                        domain = self._findDomain(sdUUID)<br>
                      File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/sdc.py",
                    line 151, in _findDomain<br>
                        return findMethod(sdUUID)<br>
                      File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/sdc.py",
                    line 176, in _findUnfetchedDomain<br>
                        raise se.StorageDomainDoesNotExist(<wbr>sdUUID)<br>
                    StorageDomainDoesNotExist: Storage domain does not
                    exist: (u'c65437ce-339f-4b01-aeb5-<wbr>45c1d486bf49',)<br>
                    2017-11-18 06:16:40,377-0500 INFO  (jsonrpc/7)
                    [api.host] START getStats()
                    from=::ffff:192.168.201.4,<wbr>58722 (api:46)<br>
                    2017-11-18 06:16:40,378-0500 INFO  (jsonrpc/7)
                    [vdsm.api] START repoStats(domains=())
                    from=::ffff:192.168.201.4,<wbr>58722,
                    task_id=8fb74944-08c0-491e-<wbr>ad55-a7a9f0a11ef8
                    (api:46)<br>
                    2017-11-18 06:16:40,379-0500 INFO  (jsonrpc/7)
                    [vdsm.api] FINISH repoStats
                    return={u'c65437ce-339f-4b01-<wbr>aeb5-45c1d486bf49':
                    {'code': 358, 'actual': True, 'version': -1,
                    'acquired': False, 'delay': '0', 'lastCheck': '6.1',
                    'valid': False},<br>
                     u'cc61e074-a3b6-4371-9185-<wbr>66079a39f123':
                    {'code': 0, 'actual': True, 'version': 4,
                    'acquired': True, 'delay': '0.00103987',
                    'lastCheck': '6.5', 'valid': True}}
                    from=::ffff:192.168.201.4,<wbr>58722,
                    task_id=8fb74944-08c0-491e-<wbr>ad55-a7a9f0a11ef8<br>
                     (api:52)<br>
                  </span></p>
                <p><span style="font-weight:normal">engine log: <br>
                  </span></p>
                <p><span style="font-weight:normal">2017-11-18
                    06:15:54,040-05 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.irsbroker.IrsProxy]
                    (EE-ManagedThreadFactory-<wbr>engine-Thread-29)
                    [4ce8aff3] Domain 'c65437ce-339f-4b01-aeb5-<wbr>45c1d486bf49:nfs'
                    was reported with error code '358'<br>
                    2017-11-18 06:15:54,041-05 ERROR
                    [org.ovirt.engine.core.bll.<wbr>InitVdsOnUpCommand]
                    (EE-ManagedThreadFactory-<wbr>engine-Thread-29)
                    [4ce8aff3] Storage Domain 'nfs' of pool 'test-dc' is
                    in problem in host 'lago-basic-suite-master-host-<wbr>1'<br>
                    2017-11-18 06:15:54,045-05 ERROR
                    [org.ovirt.engine.core.dal.<wbr>dbbroker.auditloghandling.<wbr>AuditLogDirector]
                    (EE-ManagedThreadFactory-<wbr>engine-Thread-29)
                    [4ce8aff3] EVENT_ID: VDS_STORAGE_VDS_STATS_FAILED(<wbr>189),
                    Host lago-basic-suite-master-host-1<br>
                     reports about one of the Active Storage Domains as
                    Problematic.</span><b style="font-weight:normal"
id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><b><br>
                    </b></b></p>
                <p><b style="font-weight:normal"
id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><br>
                  </b></p>
                <p dir="ltr"
                  style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><b
                    style="font-weight:normal"
id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">&lt;/error&gt;</span></b></p>
                <b style="font-weight:normal"
id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81">
                </b><br
                  class="gmail-m_-6851820411967884273Apple-interchange-newline">
                <p><br>
                </p>
              </div>
              <br>
              ______________________________<wbr>_________________<br>
              Devel mailing list<br>
              <a href="mailto:Devel@ovirt.org" moz-do-not-send="true">Devel@ovirt.org</a><br>
              <a href="http://lists.ovirt.org/mailman/listinfo/devel"
                rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br>
            </blockquote>
          </div>
          <br>
        </div>
      </div>
    </blockquote>
    <p><br>
    </p>
  </body>
</html>