<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Nov 20, 2017 at 3:10 PM, Dafna Ron <span dir="ltr">&lt;<a href="mailto:dron@redhat.com" target="_blank">dron@redhat.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
  

    
  
  <div bgcolor="#FFFFFF">
    <p>Hi, <br>
    </p>
    <p>We had a failure in OST for test
      002_bootstrap.verify_add_all_<wbr>hosts. <br>
    </p>
    <p>From the logs I can see that vdsm on host0 was reporting that it
      cannot find the physical volume but eventually the storage was
      created and is reported as responsive. <br>
    </p>
    <p>However, Host1 is reported to became non-operational with storage
      domain does not exist error and I think that there is a race. <br></p></div></blockquote><div><br></div><div>I&#39;ve opened <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1514906">https://bugzilla.redhat.com/show_bug.cgi?id=1514906</a> on this. </div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><p>
    </p>
    <p>I think that we create the storage domain while host1 is being
      installed and if the domain is not created and reported as
      activated in time, host1 will become nonOperational. <br></p></div></blockquote><div><br></div><div>And based on the above description, this is exactly the issue I&#39;ve described in the BZ.</div><div>Y.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><p>
    </p>
    <p>are we starting installation of host1 before host0 and storage
      are active? <br>
    </p>
    <p><b style="font-weight:normal" id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81">
        </b></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><b style="font-weight:normal" id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Link to suspected patches: I do not think that the patch reported is related to the error</span></b></p><b style="font-weight:normal" id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81">
        <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><b>
</b></span></p>
      </b><b style="font-weight:normal" id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81">
        <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><a class="gmail-m_-6851820411967884273moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/84133/" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>84133/</a><b>
</b></span></p>
        <br>
        <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Link to Job:</span></p>
        <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">
</span></p>
      </b><b style="font-weight:normal" id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81">
        <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><a class="gmail-m_-6851820411967884273moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3902/" target="_blank">http://jenkins.ovirt.org/job/<wbr>ovirt-master_change-queue-<wbr>tester/3902/</a>
</span></p>
        <br>
        <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Link to all logs:</span></p>
        <br>
      </b><p></p>
    <p><b style="font-weight:normal" id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><a class="gmail-m_-6851820411967884273moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3902/artifact/" target="_blank">http://jenkins.ovirt.org/job/<wbr>ovirt-master_change-queue-<wbr>tester/3902/artifact/</a><br>
      </b></p>
    <p><b style="font-weight:normal" id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><br>
        </b></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><b style="font-weight:normal" id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">(Relevant) error snippet from the log: </span></b></p><b style="font-weight:normal" id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81">
        <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">&lt;error&gt;</span></p>
        <br>
        Lago log: <br>
      </b><p></p>
    <p><span style="font-weight:normal">2017-11-18
        11:15:25,472::log_utils.py::<wbr>end_log_task::670::nose::INFO:<wbr>:  #
        add_master_storage_domain: ESC[32mSuccessESC[0m (in 0:01:09)<br>
        2017-11-18
        11:15:25,472::log_utils.py::<wbr>start_log_task::655::nose::<wbr>INFO::  #
        add_secondary_storage_domains: ESC[0mESC[0m<br>
        2017-11-18
        11:16:47,455::log_utils.py::<wbr>end_log_task::670::nose::INFO:<wbr>:  #
        add_secondary_storage_domains: ESC[32mSuccessESC[0m (in 0:01:21)<br>
        2017-11-18
        11:16:47,456::log_utils.py::<wbr>start_log_task::655::nose::<wbr>INFO::  #
        import_templates: ESC[0mESC[0m<br>
        2017-11-18
        11:16:47,513::testlib.py::<wbr>stopTest::198::nose::INFO::    *
        SKIPPED: Exported domain generation not supported yet<br>
        2017-11-18
        11:16:47,514::log_utils.py::<wbr>end_log_task::670::nose::INFO:<wbr>:  #
        import_templates: ESC[32mSuccessESC[0m (in 0:00:00)<br>
        2017-11-18
        11:16:47,514::log_utils.py::<wbr>start_log_task::655::nose::<wbr>INFO::  #
        verify_add_all_hosts: ESC[0mESC[0m<br>
        2017-11-18
11:16:47,719::testlib.py::<wbr>assert_equals_within::227::<wbr>ovirtlago.testlib::ERROR::   
        * Unhandled exception in &lt;function &lt;lambda&gt; at
        0x2909230&gt;<br>
        Traceback (most recent call last):<br>
          File &quot;/usr/lib/python2.7/site-<wbr>packages/ovirtlago/testlib.py&quot;<wbr>,
        line 219, in assert_equals_within<br>
            res = func()<br>
          File
&quot;/home/jenkins/workspace/<wbr>ovirt-master_change-queue-<wbr>tester/ovirt-system-tests/<wbr>basic-suite-master/test-<wbr>scenarios/002_bootstrap.py&quot;,
        line 430, in &lt;lambda&gt;<br>
            lambda: _all_hosts_up(hosts_service, total_hosts)<br>
          File
&quot;/home/jenkins/workspace/<wbr>ovirt-master_change-queue-<wbr>tester/ovirt-system-tests/<wbr>basic-suite-master/test-<wbr>scenarios/002_bootstrap.py&quot;,
        line 129, in _all_hosts_up<br>
            _check_problematic_hosts(<wbr>hosts_service)<br>
          File
&quot;/home/jenkins/workspace/<wbr>ovirt-master_change-queue-<wbr>tester/ovirt-system-tests/<wbr>basic-suite-master/test-<wbr>scenarios/002_bootstrap.py&quot;,
        line 149, in _check_problematic_hosts<br>
            raise RuntimeError(dump_hosts)<br>
        RuntimeError: 1 hosts failed installation:<br>
        lago-basic-suite-master-host-<wbr>1: non_operational<br>
        <br>
        2017-11-18
        11:16:47,722::utils.py::<wbr>wrapper::480::lago.utils::<wbr>DEBUG::Looking
        for a workdir<br>
        2017-11-18
11:16:47,722::workdir.py::<wbr>resolve_workdir_path::361::<wbr>lago.workdir::DEBUG::Checking
        if /dev/shm/ost/deployment-basic-<wbr>suite-master is a workdir<br>
        2017-11-18
        11:16:47,724::log_utils.py::__<wbr>enter__::600::lago.prefix::<wbr>INFO::   
        * Collect artifacts: ESC[0mESC[0m<br>
        2017-11-18
        11:16:47,724::log_utils.py::__<wbr>enter__::600::lago.prefix::<wbr>INFO::   
        * Collect artifacts: ESC[0mESC[0m<br>
      </span></p>
    <p><span style="font-weight:normal">vdsm host0: <br>
      </span></p>
    <p><span style="font-weight:normal">2017-11-18 06:14:23,980-0500
        INFO  (jsonrpc/0) [vdsm.api] START getDeviceList(storageType=3,
        guids=[u&#39;<wbr>360014059618895272774e97a2aaf5<wbr>dd6&#39;], checkStatus=False,
        options={}) from=::ffff:192.168.201.4,<wbr>45636,
        flow_id=ed8310a1-a7af-4a67-<wbr>b351-8ff<br>
        364766b8a, task_id=6ced0092-34cd-49f0-<wbr>aa0f-6aae498af37f (api:46)<br>
        2017-11-18 06:14:24,353-0500 WARN  (jsonrpc/0) [storage.LVM] lvm
        pvs failed: 5 [] [&#39;  Failed to find physical volume
        &quot;/dev/mapper/<wbr>360014059618895272774e97a2aaf5<wbr>dd6&quot;.&#39;] (lvm:322)<br>
        2017-11-18 06:14:24,353-0500 WARN  (jsonrpc/0) [storage.HSM]
        getPV failed for guid: 360014059618895272774e97a2aaf5<wbr>dd6
        (hsm:1973)<br>
        Traceback (most recent call last):<br>
          File &quot;/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/hsm.py&quot;,
        line 1970, in _getDeviceList<br>
            pv = lvm.getPV(guid)<br>
          File &quot;/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/lvm.py&quot;,
        line 852, in getPV<br>
            raise se.InaccessiblePhysDev((<wbr>pvName,))<br>
        InaccessiblePhysDev: Multipath cannot access physical device(s):
        &quot;devices=(u&#39;<wbr>360014059618895272774e97a2aaf5<wbr>dd6&#39;,)&quot;<br>
        2017-11-18 06:14:24,389-0500 INFO  (jsonrpc/0) [vdsm.api] FINISH
        getDeviceList return={&#39;devList&#39;: [{&#39;status&#39;: &#39;unknown&#39;,
        &#39;vendorID&#39;: &#39;LIO-ORG&#39;, &#39;capacity&#39;: &#39;21474836480&#39;, &#39;fwrev&#39;:
        &#39;4.0&#39;, &#39;discard_zeroes_data&#39;: 0, &#39;vgUUID&#39;: &#39;&#39;, &#39;pvsize&#39;: &#39;&#39;,
        &#39;pathlist&#39;: [{&#39;initiatorname&#39;: u&#39;default&#39;, &#39;connection&#39;:
        u&#39;192.168.200.4&#39;, &#39;iqn&#39;: u&#39;iqn.2014-07.org.ovirt:<wbr>storage&#39;,
        &#39;portal&#39;: &#39;1&#39;, &#39;user&#39;: u&#39;username&#39;, &#39;password&#39;: &#39;********&#39;,
        &#39;port&#39;: &#39;3260&#39;}, {&#39;initiatorname&#39;: u&#39;default&#39;, &#39;connection&#39;:
        u&#39;192.168.201.4&#39;, &#39;iqn&#39;: u&#39;iqn.2014-07.org.ovirt:<wbr>storage&#39;,
        &#39;portal&#39;: &#39;1&#39;, &#39;user&#39;: u&#39;username&#39;, &#39;password&#39;: &#39;********&#39;,
        &#39;port&#39;: &#39;3260&#39;}], &#39;logicalblocksize&#39;: &#39;512&#39;,
        &#39;discard_max_bytes&#39;: 1073741824, &#39;pathstatus&#39;: [{&#39;type&#39;:
        &#39;iSCSI&#39;, &#39;physdev&#39;: &#39;sda&#39;, &#39;capacity&#39;: &#39;21474836480&#39;, &#39;state&#39;:
        &#39;active&#39;, &#39;lun&#39;: &#39;0&#39;}, {&#39;type&#39;: &#39;iSCSI&#39;, &#39;physdev&#39;: &#39;sdf&#39;,
        &#39;capacity&#39;: &#39;21474836480&#39;, &#39;state&#39;: &#39;active&#39;, &#39;lun&#39;: &#39;0&#39;}],
        &#39;devtype&#39;: &#39;iSCSI&#39;, &#39;physicalblocksize&#39;: &#39;512&#39;, &#39;pvUUID&#39;: &#39;&#39;,
        &#39;serial&#39;:
        &#39;SLIO-ORG_lun0_bdev_96188952-<wbr>7277-4e97-a2aa-f5dd6aad6fc2&#39;,
        &#39;GUID&#39;: &#39;<wbr>360014059618895272774e97a2aaf5<wbr>dd6&#39;, &#39;productID&#39;:
        &#39;lun0_bdev&#39;}]} from=::ffff:192.168.201.4,<wbr>45636,
        flow_id=ed8310a1-a7af-4a67-<wbr>b351-8ff364766b8a,
        task_id=6ced0092-34cd-49f0-<wbr>aa0f-6aae498af37f (api:52)</span></p>
    <p><span style="font-weight:normal"><br>
      </span></p>
    <p><span style="font-weight:normal">2017-11-18 06:14:31,788-0500
        INFO  (jsonrpc/0) [vdsm.api] FINISH getStorageDomainInfo
        return={&#39;info&#39;: {&#39;uuid&#39;: &#39;cc61e074-a3b6-4371-9185-<wbr>66079a39f123&#39;,
        &#39;vgMetadataDevice&#39;: &#39;<wbr>360014059618895272774e97a2aaf5<wbr>dd6&#39;,
        &#39;vguuid&#39;: &#39;7ifbmt-0elj-uWZZ-zS<br>
        LG-plA8-8hd3-JG298b&#39;, &#39;metadataDevice&#39;:
        &#39;<wbr>360014059618895272774e97a2aaf5<wbr>dd6&#39;, &#39;state&#39;: &#39;OK&#39;, &#39;version&#39;:
        &#39;4&#39;, &#39;role&#39;: &#39;Regular&#39;, &#39;type&#39;: &#39;ISCSI&#39;, &#39;class&#39;: &#39;Data&#39;,
        &#39;pool&#39;: [], &#39;name&#39;: &#39;iscsi&#39;}} from=::ffff:192.168.201.4,<wbr>45636,
        flow_id=2c1876<br>
        99, task_id=c2080b61-d4a5-4bdb-<wbr>9d75-f81580a8257a (api:<br>
      </span></p>
    <p><span style="font-weight:normal">vdsm host1:</span></p>
    <p><span style="font-weight:normal">2017-11-18 06:16:34,315-0500
        ERROR (monitor/c65437c) [storage.Monitor] Setting up monitor for
        c65437ce-339f-4b01-aeb5-<wbr>45c1d486bf49 failed (monitor:329)<br>
        Traceback (most recent call last):<br>
          File
        &quot;/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/monitor.<wbr>py&quot;, line
        326, in _setupLoop<br>
            self._setupMonitor()<br>
          File
        &quot;/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/monitor.<wbr>py&quot;, line
        348, in _setupMonitor<br>
            self._produceDomain()<br>
          File &quot;/usr/lib/python2.7/site-<wbr>packages/vdsm/utils.py&quot;, line
        177, in wrapper<br>
            value = meth(self, *a, **kw)<br>
          File
        &quot;/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/monitor.<wbr>py&quot;, line
        366, in _produceDomain<br>
            self.domain = sdCache.produce(self.sdUUID)<br>
          File &quot;/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/sdc.py&quot;,
        line 110, in produce<br>
            domain.getRealDomain()<br>
          File &quot;/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/sdc.py&quot;,
        line 51, in getRealDomain<br>
            return self._cache._realProduce(self.<wbr>_sdUUID)<br>
          File &quot;/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/sdc.py&quot;,
        line 134, in _realProduce<br>
            domain = self._findDomain(sdUUID)<br>
          File &quot;/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/sdc.py&quot;,
        line 151, in _findDomain<br>
            return findMethod(sdUUID)<br>
          File &quot;/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/sdc.py&quot;,
        line 176, in _findUnfetchedDomain<br>
            raise se.StorageDomainDoesNotExist(<wbr>sdUUID)<br>
        StorageDomainDoesNotExist: Storage domain does not exist:
        (u&#39;c65437ce-339f-4b01-aeb5-<wbr>45c1d486bf49&#39;,)<br>
        2017-11-18 06:16:40,377-0500 INFO  (jsonrpc/7) [api.host] START
        getStats() from=::ffff:192.168.201.4,<wbr>58722 (api:46)<br>
        2017-11-18 06:16:40,378-0500 INFO  (jsonrpc/7) [vdsm.api] START
        repoStats(domains=()) from=::ffff:192.168.201.4,<wbr>58722,
        task_id=8fb74944-08c0-491e-<wbr>ad55-a7a9f0a11ef8 (api:46)<br>
        2017-11-18 06:16:40,379-0500 INFO  (jsonrpc/7) [vdsm.api] FINISH
        repoStats return={u&#39;c65437ce-339f-4b01-<wbr>aeb5-45c1d486bf49&#39;:
        {&#39;code&#39;: 358, &#39;actual&#39;: True, &#39;version&#39;: -1, &#39;acquired&#39;: False,
        &#39;delay&#39;: &#39;0&#39;, &#39;lastCheck&#39;: &#39;6.1&#39;, &#39;valid&#39;: False},<br>
         u&#39;cc61e074-a3b6-4371-9185-<wbr>66079a39f123&#39;: {&#39;code&#39;: 0, &#39;actual&#39;:
        True, &#39;version&#39;: 4, &#39;acquired&#39;: True, &#39;delay&#39;: &#39;0.00103987&#39;,
        &#39;lastCheck&#39;: &#39;6.5&#39;, &#39;valid&#39;: True}}
        from=::ffff:192.168.201.4,<wbr>58722,
        task_id=8fb74944-08c0-491e-<wbr>ad55-a7a9f0a11ef8<br>
         (api:52)<br>
      </span></p>
    <p><span style="font-weight:normal">engine log: <br>
      </span></p>
    <p><span style="font-weight:normal">2017-11-18 06:15:54,040-05
        ERROR [org.ovirt.engine.core.<wbr>vdsbroker.irsbroker.IrsProxy]
        (EE-ManagedThreadFactory-<wbr>engine-Thread-29) [4ce8aff3] Domain
        &#39;c65437ce-339f-4b01-aeb5-<wbr>45c1d486bf49:nfs&#39; was reported with
        error code &#39;358&#39;<br>
        2017-11-18 06:15:54,041-05 ERROR
        [org.ovirt.engine.core.bll.<wbr>InitVdsOnUpCommand]
        (EE-ManagedThreadFactory-<wbr>engine-Thread-29) [4ce8aff3] Storage
        Domain &#39;nfs&#39; of pool &#39;test-dc&#39; is in problem in host
        &#39;lago-basic-suite-master-host-<wbr>1&#39;<br>
        2017-11-18 06:15:54,045-05 ERROR
        [org.ovirt.engine.core.dal.<wbr>dbbroker.auditloghandling.<wbr>AuditLogDirector]
        (EE-ManagedThreadFactory-<wbr>engine-Thread-29) [4ce8aff3] EVENT_ID:
        VDS_STORAGE_VDS_STATS_FAILED(<wbr>189), Host
        lago-basic-suite-master-host-1<br>
         reports about one of the Active Storage Domains as Problematic.</span><b style="font-weight:normal" id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><b><br>
        </b></b></p>
    <p><b style="font-weight:normal" id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><b></b><br>
        </b></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><b style="font-weight:normal" id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">&lt;/error&gt;</span></b></p><b style="font-weight:normal" id="gmail-m_-6851820411967884273docs-internal-guid-5859b7a1-d974-a2c9-3d0d-cb5378c92f81">
      </b><br class="gmail-m_-6851820411967884273Apple-interchange-newline">
    <p></p>
    <p><br>
    </p>
  </div>

<br>______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br></blockquote></div><br></div></div>